Nov 03 20:03:05.400904 ip-10-0-136-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:03:05.797156 ip-10-0-136-244 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:05.797156 ip-10-0-136-244 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:03:05.797156 ip-10-0-136-244 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:05.797156 ip-10-0-136-244 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:05.800996 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.800873    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.800996 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.800996    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.800996 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801002    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801005    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801008    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801011    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801015    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801018    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801021    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801024    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801027    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801029    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801032    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801034    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801037    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801040    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801042    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801045    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801047    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801050    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801058    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801061    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.801088 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801064    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801067    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801069    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801072    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801074    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801076    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801079    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801082    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801084    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801087    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801089    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801092    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801095    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801097    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801100    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801103    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801105    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801108    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801111    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.801587 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801113    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801115    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801117    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801122    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801126    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801129    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801132    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801135    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801137    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801140    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801143    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801145    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801147    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801152    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801155    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801158    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801161    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801163    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.802041 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801166    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801171    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801175    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801178    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801181    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801185    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801188    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801190    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801193    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801195    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801198    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801200    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801203    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801205    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801208    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801210    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801213    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801215    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801218    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.802492 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801220    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801223    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801226    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801229    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801232    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801234    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801237    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801239    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801242    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801245    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801247    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801250    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801252    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801255    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801258    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801260    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801263    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801265    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801267    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801270    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.802959 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801272    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801274    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801277    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801280    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801282    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801286    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801288    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.801290    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802115    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802122    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802126    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802129    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802132    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802134    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802138    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802140    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802143    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802146    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802148    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.803442 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802151    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802153    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802156    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802159    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802161    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802164    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802166    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802169    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802172    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802174    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802178    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802180    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802183    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802185    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802188    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802190    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802193    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802195    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802198    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802200    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.803910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802203    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802205    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802208    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802211    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802213    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802216    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802219    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802222    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802226    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802228    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802231    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802235    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802239    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802242    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802245    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802249    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802252    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802254    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802256    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.804378 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802259    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802262    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802264    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802267    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802269    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802271    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802275    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802278    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802280    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802283    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802285    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802288    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802290    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802293    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802295    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802298    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802301    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802304    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802306    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.804835 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802309    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802312    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802314    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802317    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802319    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802322    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802324    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802327    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802329    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802332    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802334    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802337    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802339    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802341    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802344    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802346    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802348    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802351    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802353    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802356    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.805277 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802358    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802361    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802363    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802365    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802368    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802370    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802373    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802375    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802380    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802382    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802386    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802388    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802391    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802394    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802396    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802398    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802401    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802914    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802919    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802922    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.805801 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802925    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802927    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802930    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802932    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802935    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802937    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802940    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802942    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802945    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802947    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802950    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802952    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802955    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802958    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802961    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802963    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802966    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802969    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802971    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.806286 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802974    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802977    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802980    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802983    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802985    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802987    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802990    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802993    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802995    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.802999    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803001    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803004    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803006    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803009    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803011    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803014    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803016    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803018    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803021    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803023    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.806741 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803026    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803028    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803030    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803033    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803035    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803038    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803040    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803043    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803045    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803050    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803052    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803055    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803057    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803060    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803063    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803065    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803067    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803070    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803073    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.807211 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803075    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803078    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803080    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803083    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803086    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803088    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803091    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803093    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803095    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803098    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803100    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803103    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803105    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803107    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803110    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803112    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803114    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803117    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803119    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.807692 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803124    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803128    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803131    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803135    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803139    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803141    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803144    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803146    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803149    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803152    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803154    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803157    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803159    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803162    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803165    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803169    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803172    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803175    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803177    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803180    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.808135 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803182    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803185    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803187    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803190    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803193    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.803195    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803968    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803978    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803986    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803990    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803995    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.803999    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804003    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804007    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804011    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804014    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804018    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804021    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804024    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804027    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804030    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804033    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804036    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:03:05.808620 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804039    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804045    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804048    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804051    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804054    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804057    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804061    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804065    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804068    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804071    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804074    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804082    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804085    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804088    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804092    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804096    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804099    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804102    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804105    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804108    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804111    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804115    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804118    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804121    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:03:05.809200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804124    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804127    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804133    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804137    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804140    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804143    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804146    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804149    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804152    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804155    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804158    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804160    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804163    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804168    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804171    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804174    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804179    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804182    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804185    2575 flags.go:64] FLAG: --help="false"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804188    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-136-244.ec2.internal"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804191    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804194    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804197    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804200    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:03:05.809920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804204    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804207    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804209    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804212    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804215    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804218    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804221    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804224    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804227    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804229    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804236    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804239    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804242    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804245    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804248    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804251    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804257    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804260    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804263    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804266    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804268    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804272    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804275    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804278    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804282    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:03:05.810489 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804286    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804290    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804294    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804297    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804300    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804303    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804306    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804308    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804311    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804319    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804322    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804325    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804328    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804331    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804334    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804337    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804340    2575 flags.go:64] FLAG: --port="10250"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804345    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804348    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089ab518f32319645"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804351    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804354    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804357    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804360    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804364    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804366    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804369    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:03:05.811111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804372    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804376    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804379    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804382    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804385    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804388    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804391    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804394    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804398    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804400    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804403    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804406    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804409    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804415    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804418    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804421    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804424    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804427    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804430    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804433    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804435    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804441    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804444    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804448    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804453    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:03:05.811744 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804455    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804458    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804461    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804464    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804467    2575 flags.go:64] FLAG: --v="2"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804472    2575 flags.go:64] FLAG: --version="false"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804476    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804480    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.804483    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804647    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804652    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804655    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804657    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804660    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804663    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804666    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804669    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804672    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804674    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804677    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804681    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.812331 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804684    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804687    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804689    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804692    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804694    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804697    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804699    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804702    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804705    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804708    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804711    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804713    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804716    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804718    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804721    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804723    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804726    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804728    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804731    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.812886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804733    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804735    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804738    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804740    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804743    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804745    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804748    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804750    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804753    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804756    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804758    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804761    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804765    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804769    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804772    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804775    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804779    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804783    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804786    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804790    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.813345 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804792    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804795    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804799    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804801    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804806    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804809    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804811    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804814    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804816    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804819    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804821    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804826    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804828    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804831    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804833    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804836    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804839    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804841    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.813886 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804844    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804847    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804850    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804852    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804856    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804859    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804861    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804864    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804866    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.804869    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805236    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805240    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805242    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805245    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805249    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805252    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805254    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805257    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805259    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805262    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.814358 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805264    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805267    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805269    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805272    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805275    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805277    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805280    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805283    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805285    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805288    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805291    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805293    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805296    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805298    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805302    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.805330    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:05.814852 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805375    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805380    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805383    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805386    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805389    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805392    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805395    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805398    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805400    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805403    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805405    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805409    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805412    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805415    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805418    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805420    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805423    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805425    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805428    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805431    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.815233 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805433    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805436    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805439    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805441    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805443    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805446    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805448    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805451    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805456    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805459    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805462    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805466    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805468    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805471    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805473    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805476    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805478    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805480    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.815708 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805483    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805485    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805488    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805490    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805494    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805496    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805499    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805501    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805504    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805506    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805509    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805511    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805514    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805516    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805519    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805521    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805524    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805540    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805543    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805545    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.816173 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805548    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805550    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805553    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805557    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805560    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805565    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805569    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805572    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805575    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805578    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805580    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805583    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805585    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805588    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805591    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805594    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805597    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805600    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805603    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.816652 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805605    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805608    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805610    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805613    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805615    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805618    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805620    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805623    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805625    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805628    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805630    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805632    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805635    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805637    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805640    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805643    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805646    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805649    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805652    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805655    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.817108 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805658    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805660    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805663    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805665    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805668    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805670    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805673    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.805676    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.812754    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.812771    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.813713    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.813720    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813801    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813807    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813812    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813817    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813821    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813824    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813827    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.817599 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813830    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813833    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813835    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813838    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813841    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813843    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813846    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813849    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813852    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813854    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813857    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813859    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813862    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813865    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813867    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813870    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813873    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813875    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813878    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813880    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.818080 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813883    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813885    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813888    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813891    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813894    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813897    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813899    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813902    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813905    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813907    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813911    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813913    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813916    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813918    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813921    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813923    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813926    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813929    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813931    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.818554 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813934    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813936    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813939    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813941    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813944    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813946    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813949    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813952    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813955    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813957    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813960    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813962    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813965    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813967    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813969    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813973    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813976    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813978    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813981    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813984    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.819006 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813986    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813989    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813992    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813994    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813997    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.813999    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814001    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814004    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814006    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814009    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814011    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814014    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814016    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814019    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814021    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814024    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814026    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814028    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814031    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814033    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.819472 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814037    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814039    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814042    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814044    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814047    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814049    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814052    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814055    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814058    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814060    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814062    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814065    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814068    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814070    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814073    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814076    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814080    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814082    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814085    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.819945 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814088    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.814122    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814270    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814275    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814278    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814281    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814284    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814286    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814289    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814291    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814294    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814296    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814298    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814301    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814304    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814306    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.820379 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814309    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814311    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814314    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814316    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814319    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814322    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814324    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814327    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814329    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814332    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814334    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814337    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814340    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814342    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814344    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814347    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814349    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814351    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814354    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814357    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.820768 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814359    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814362    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814365    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814367    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814370    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814373    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814375    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814378    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814381    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814383    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814386    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814389    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814392    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814394    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814397    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814399    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814403    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814407    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814411    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.821255 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814414    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814417    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814421    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814423    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814426    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814429    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814431    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814433    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814436    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814438    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814441    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814443    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814446    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814448    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814450    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814453    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814455    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814458    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814460    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.822091 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814463    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814465    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814468    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814470    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814473    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814475    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814478    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814481    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814483    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814486    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814489    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814491    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814494    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814496    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814498    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814501    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814503    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814506    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814508    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.822717 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814510    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814513    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814515    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814518    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814520    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814523    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814541    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814543    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814546    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814549    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814551    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814554    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814556    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814559    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814561    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.814585    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:05.823205 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814619    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814623    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814626    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814629    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814633    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814636    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814639    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814642    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814644    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814648    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814651    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814653    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814657    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814660    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814662    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814665    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814668    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814670    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814673    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814675    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:05.823646 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814678    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814680    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814683    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814686    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814688    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814691    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814693    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814695    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814698    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814701    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814703    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814706    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814709    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814711    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814713    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814716    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814719    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814722    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814724    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814726    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:05.824164 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814729    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814732    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814734    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814736    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814739    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814741    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814743    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814746    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814748    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814751    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814753    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814756    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814758    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814760    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814765    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814767    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814770    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814772    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814775    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814777    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:05.824736 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814781    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814784    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814786    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814789    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814791    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814793    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814796    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814799    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814801    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814804    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814807    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814809    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814812    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814814    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814817    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814820    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814823    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814825    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814828    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:05.825265 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814830    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814832    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814835    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814838    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814840    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814843    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814845    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814849    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814852    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814854    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814857    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814860    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814863    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814868    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814872    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814875    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814879    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814883    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814886    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:05.825766 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814889    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814892    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814894    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814897    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814900    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814903    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814906    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:05.814909    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.814967    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.817389    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.818132    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.818234    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.818872    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:03:05.826218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.819891    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:03:05.826661 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.826553    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:03:05.838000 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.837980    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:03:05.841722 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.841702    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 20:03:05.842590 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.842558    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:03:05.845746 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.845710    2575 fs.go:135] Filesystem UUIDs: map[73a26d9b-d44c-406a-b5fd-bff4812995a6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d113a559-edde-4a98-a7a8-a4c307836dde:/dev/nvme0n1p4]
Nov 03 20:03:05.845840 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.845743    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:03:05.850846 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.850718    2575 manager.go:217] Machine: {Timestamp:2026-11-03 20:03:05.849505739 +0000 UTC m=+0.333576311 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3246939 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2808e94028c5ba20b667f4c8a14ad3 SystemUUID:ec2808e9-4028-c5ba-20b6-67f4c8a14ad3 BootID:2967ee86-3264-4b87-a319-b32ad7f4420f Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:f7:b8:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:f7:b8:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:37:cb:a2:9d:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:03:05.850846 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.850838    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:03:05.851011 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.850932    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:03:05.853898 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.853867    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:03:05.854055 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.853899    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:03:05.854107 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.854057    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:03:05.854107 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.854066    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:03:05.854160 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.854115    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:03:05.854160 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.854135    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:03:05.854915 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.854903    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:03:05.857617 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.857603    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:03:05.857748 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.857738    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:03:05.858378 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858369    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:03:05.858413 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858384    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:03:05.858413 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858396    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:03:05.858413 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858405    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:03:05.858413 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858413    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:03:05.858820 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.858808    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:03:05.860882 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.860855    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:03:05.865188 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.865173    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:03:05.866063 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866050    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:03:05.866100 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866082    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:03:05.866630 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866619    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:03:05.866673 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866636    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:03:05.866673 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866643    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:03:05.866673 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866649    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:03:05.866673 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866654    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:03:05.866673 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.866661    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:03:05.867136 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867128    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:03:05.867165 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867137    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:03:05.867165 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867144    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:03:05.867165 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867150    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:03:05.867165 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867160    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:03:05.867291 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.867168    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:03:05.868414 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.868403    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:03:05.868452 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.868418    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:03:05.872330 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.872313    2575 server.go:1263] "Started kubelet"
Nov 03 20:03:05.872688 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.872575    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:03:05.872688 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.872516    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:03:05.872688 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.872644    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:03:05.875606 ip-10-0-136-244 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:03:05.878042 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.878022    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:03:05.878962 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.878939    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:03:05.880182 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.880156    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:03:05.883217 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.883200    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:03:05.883667 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.883649    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:03:05.883848 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.883828    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:05.884212 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884187    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:03:05.884286 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884190    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:03:05.884286 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884233    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:03:05.884393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884362    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:03:05.884393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884370    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:03:05.884506 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884489    2575 factory.go:55] Registering systemd factory
Nov 03 20:03:05.884612 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884509    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:03:05.884826 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884776    2575 factory.go:153] Registering CRI-O factory
Nov 03 20:03:05.884826 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884791    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:03:05.884933 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884840    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:03:05.884933 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884857    2575 factory.go:103] Registering Raw factory
Nov 03 20:03:05.884933 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.884868    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:03:05.885829 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.885813    2575 manager.go:319] Starting recovery of all containers
Nov 03 20:03:05.897019 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.896997    2575 manager.go:324] Recovery completed
Nov 03 20:03:05.901621 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.901601    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:05.902724 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.902674    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:03:05.902724 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.902699    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:03:05.903547 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.902678    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-244.ec2.internal.18e4a18eb3ae7406  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-244.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-244.ec2.internal,},FirstTimestamp:2026-11-03 20:03:05.87228263 +0000 UTC m=+0.356353201,LastTimestamp:2026-11-03 20:03:05.87228263 +0000 UTC m=+0.356353201,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-244.ec2.internal,}"
Nov 03 20:03:05.903657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.903630    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:05.903705 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.903680    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:05.903705 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.903691    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:05.904216 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.904199    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:03:05.904216 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.904211    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:03:05.904216 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.904216    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:05.904353 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.904230    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:03:05.905749 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.905723    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:05.905820 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.905765    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:05.905820 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.905780    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:05.906021 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.906003    2575 policy_none.go:50] "Start"
Nov 03 20:03:05.906120 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.906033    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:03:05.906120 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.906050    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:03:05.907827 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.907489    2575 policy_none.go:44] "Start"
Nov 03 20:03:05.910416 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.910261    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:03:05.910509 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.910335    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8w2rn"
Nov 03 20:03:05.910509 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.910474    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8w2rn"
Nov 03 20:03:05.928039 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.928010    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 20:03:05.928203 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.928126    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 20:03:05.928247 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.928216    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 20:03:05.928247 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.928170    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-244.ec2.internal.18e4a18eb58d410a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-244.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-244.ec2.internal,},FirstTimestamp:2026-11-03 20:03:05.903661322 +0000 UTC m=+0.387731900,LastTimestamp:2026-11-03 20:03:05.903661322 +0000 UTC m=+0.387731900,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-244.ec2.internal,}"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.957554    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.958942    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.958965    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.958988    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.959033    2575 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:03:05.962835 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.962407    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:03:05.965197 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965180    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:03:05.965314 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.965210    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:03:05.965314 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965219    2575 server.go:87] "Starting device plugin registration server"
Nov 03 20:03:05.965522 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965508    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:03:05.965617 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965522    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:03:05.965671 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965642    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:03:05.965760 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965741    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:03:05.965760 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.965757    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:03:05.966310 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.966288    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:03:05.966419 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:05.966403    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:05.973161 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:05.973144    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:06.059588 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.059488    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"]
Nov 03 20:03:06.059724 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.059614    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:06.060786 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.060766    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:06.060882 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.060806    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:06.060882 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.060820    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:06.062512 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.062494    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:06.062666 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.062650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.062718 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.062681    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:06.063386 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063371    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:06.063456 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063401    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:06.063456 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063417    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:06.063456 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063423    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:06.063456 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063444    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:06.063456 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.063453    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:06.064873 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.064858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.064928 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.064885    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:06.065632 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.065605    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:06.065750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.065649    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:06.065750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.065677    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:06.065750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.065702    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:06.066305 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.066279    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:06.066393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.066312    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:06.066393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.066329    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:06.066393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.066357    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.084602 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.084575    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.084602 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.084606    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-244.ec2.internal\": node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.085310 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.085291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.085347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.085317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.085347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.085335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.095635 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.095604    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-244.ec2.internal\" not found" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.099737 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.099713    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-244.ec2.internal\" not found" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.121490 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.121453    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.185708 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.185708 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.185938 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.185938 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185796    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.185938 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185803    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.185938 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.185797    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.222110 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.222075    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.323052 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.322997    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.399575 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.399545    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.402820 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.402796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:06.424082 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.424051    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.524632 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.524589    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.625155 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.625064    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.725641 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.725592    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.818139 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.818108    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:03:06.818694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.818286    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:03:06.826483 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.826433    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.884027 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:06.883946    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a01309248349c50ad695700872f4edd.slice/crio-9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1 WatchSource:0}: Error finding container 9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1: Status 404 returned error can't find the container with id 9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1
Nov 03 20:03:06.889295 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.889272    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:03:06.889395 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:06.889371    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e8a73df7412882cc22060fa0198ede.slice/crio-1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf WatchSource:0}: Error finding container 1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf: Status 404 returned error can't find the container with id 1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf
Nov 03 20:03:06.914305 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.914259    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:58:06 +0000 UTC" deadline="2028-07-08 12:15:32.626499925 +0000 UTC"
Nov 03 20:03:06.914305 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.914295    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14704h12m25.712208473s"
Nov 03 20:03:06.926826 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:06.926790    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:03:06.963058 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.963007    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerStarted","Data":"1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf"}
Nov 03 20:03:06.964056 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.964036    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal" event={"ID":"7a01309248349c50ad695700872f4edd","Type":"ContainerStarted","Data":"9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1"}
Nov 03 20:03:06.985371 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:06.985348    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:07.074667 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.074635    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:07.084837 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.084814    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:07.106346 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.106308    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:03:07.108628 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.108611    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:03:07.127570 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.127543    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:03:07.535998 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.535945    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:07.724648 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.724552    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:07.860676 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.860640    2575 apiserver.go:52] "Watching apiserver"
Nov 03 20:03:07.884113 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.884076    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:03:07.889556 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.889513    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:03:07.889987 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.889962    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7mnpc","kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal","openshift-cluster-node-tuning-operator/tuned-d4ll8","openshift-multus/multus-additional-cni-plugins-h8tqh","openshift-dns/node-resolver-c4jcr","openshift-image-registry/node-ca-t4x2d","openshift-multus/multus-nbmrq","openshift-multus/network-metrics-daemon-th9cw","openshift-ovn-kubernetes/ovnkube-node-vz4pd","kube-system/konnectivity-agent-jqq6k","openshift-network-diagnostics/network-check-target-5vbqn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal","openshift-network-operator/iptables-alerter-zgrz9"]
Nov 03 20:03:07.891787 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.891755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:07.891922 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:07.891858    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:07.893182 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.893159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.894589 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.894567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:07.894695 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:07.894649    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:07.897015 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.896994    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.898729 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.898708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.900146 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.900123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:07.901572 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.901525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:07.901672 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:07.901617    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:07.903103 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.903081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:07.905587 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.905093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.906889 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.906375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:07.909044 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.909026    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\""
Nov 03 20:03:07.909128 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.909088    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:03:07.909187 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.909154    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:03:07.909236 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.909224    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:03:07.909904 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.909877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.912360 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.912339    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:07.942747 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.942714    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:03:07.942901 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.942828    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j5slv"
Nov 03 20:03:07.985691 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.985657    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:03:07.996958 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.996919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.997136 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.996968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2lt\" (UniqueName: \"kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:07.997136 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.996998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktb47\" (UniqueName: \"kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:07.997136 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.997136 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:07.997347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:07.997347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.997347 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qn5r\" (UniqueName: \"kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxmr\" (UniqueName: \"kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:07.997886 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.997886 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:07.997886 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.997886 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:07.997886 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998144 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.997912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998144 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998144 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.998144 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.998342 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:07.998342 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998342 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998342 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.998541 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.998541 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.998541 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjwr\" (UniqueName: \"kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.998541 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998754 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998754 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998754 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998754 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998754 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998839    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998895    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.998961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.998984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999085    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bght\" (UniqueName: \"kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.999271 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn2f\" (UniqueName: \"kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:07.999641 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t66m\" (UniqueName: \"kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:07.999980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.000007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmng\" (UniqueName: \"kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.000041 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.000031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.000579 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.000055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.000579 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.000145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.018088 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018049    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.018314 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018172    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.018625 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018599    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:03:08.018831 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018808    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\""
Nov 03 20:03:08.018915 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018854    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.019037 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.018996    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:03:08.019108 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019068    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.019108 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019098    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:03:08.019270 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019247    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:03:08.019367 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019352    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.019442 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019386    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.019442 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019404    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:03:08.019442 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019396    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.019442 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019411    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.019442 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019441    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019454    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019486    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019507    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019521    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019561    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019669    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\""
Nov 03 20:03:08.019737 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019738    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\""
Nov 03 20:03:08.020096 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019787    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\""
Nov 03 20:03:08.020096 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.019997    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:03:08.020198 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020095    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:03:08.020198 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020171    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\""
Nov 03 20:03:08.020308 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020263    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\""
Nov 03 20:03:08.020397 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020355    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\""
Nov 03 20:03:08.020460 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020431    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\""
Nov 03 20:03:08.020521 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020458    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.020521 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020485    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:03:08.020655 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.020618    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100496    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmng\" (UniqueName: \"kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2lt\" (UniqueName: \"kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100654    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb47\" (UniqueName: \"kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100695    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.104674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qn5r\" (UniqueName: \"kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxmr\" (UniqueName: \"kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.100991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.105431 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjwr\" (UniqueName: \"kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101292    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101309    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101357    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106050 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101567    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101714    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101743    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101772    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101749    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101778    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101801    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101812    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101823    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101837    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.106694 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101829    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101858    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101871    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101901    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101925    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101930    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101934    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101973    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102017    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102061    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102063    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102102    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102174    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bght\" (UniqueName: \"kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102235    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:08.107439 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn2f\" (UniqueName: \"kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102540    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102540    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.101722    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.102655    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102706    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102720    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108104 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.102745    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:08.602713118 +0000 UTC m=+3.086783696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102775    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102802    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102828    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102830    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.102903    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102918    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.102953    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:08.602935497 +0000 UTC m=+3.087006059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102970    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.102999    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t66m\" (UniqueName: \"kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103035    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.108871 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103068    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103109    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103141    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103192    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103369    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103542    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103553    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.109533 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103675    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103740    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103837    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.103901    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104162    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104281    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104395    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104640    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.104802    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.105291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.106189    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.106572    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.110167 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.107161    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:08.121913 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.121230    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:08.121913 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.121261    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:08.121913 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.121279    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:08.121913 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.121412    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:08.621388026 +0000 UTC m=+3.105458604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:08.123345 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.123308    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb47\" (UniqueName: \"kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.125640 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.125434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjwr\" (UniqueName: \"kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.127127 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.126358    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bght\" (UniqueName: \"kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.127127 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.126678    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxmr\" (UniqueName: \"kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.127127 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.126831    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn2f\" (UniqueName: \"kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.127408 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.127307    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2lt\" (UniqueName: \"kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:08.127780 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.127753    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmng\" (UniqueName: \"kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.128310 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.128201    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qn5r\" (UniqueName: \"kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.130195 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.130168    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t66m\" (UniqueName: \"kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.212285 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.212247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:03:08.225082 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.225043    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49fbaa8_c462_47e9_8cf7_9b20d42180e8.slice/crio-0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb WatchSource:0}: Error finding container 0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb: Status 404 returned error can't find the container with id 0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb
Nov 03 20:03:08.233936 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.233909    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:03:08.236467 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.236375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nbmrq"
Nov 03 20:03:08.244249 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.244211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:03:08.245609 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.245582    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2414e39_9288_425f_9007_319e0815c130.slice/crio-9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6 WatchSource:0}: Error finding container 9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6: Status 404 returned error can't find the container with id 9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6
Nov 03 20:03:08.251299 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.251245    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf063b36a_960c_4ed2_bcc5_d92e7268a094.slice/crio-baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2 WatchSource:0}: Error finding container baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2: Status 404 returned error can't find the container with id baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2
Nov 03 20:03:08.255075 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.255022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:03:08.256832 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.256807    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9853ca_6edf_4ccb_b311_e2f57137b882.slice/crio-7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e WatchSource:0}: Error finding container 7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e: Status 404 returned error can't find the container with id 7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e
Nov 03 20:03:08.268910 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.268881    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbb7cff_dc60_4331_834b_416dd7d685a5.slice/crio-62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8 WatchSource:0}: Error finding container 62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8: Status 404 returned error can't find the container with id 62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8
Nov 03 20:03:08.269831 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.269810    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:08.271065 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.271016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:08.277931 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.277903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:03:08.281486 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.281453    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3830e1_66fd_4ea7_a6ba_1800e3fbaf73.slice/crio-5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6 WatchSource:0}: Error finding container 5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6: Status 404 returned error can't find the container with id 5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6
Nov 03 20:03:08.285373 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.285349    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:03:08.289838 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:08.289813    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f983d9e_13b0_40a1_99a9_8e31927ae731.slice/crio-5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1 WatchSource:0}: Error finding container 5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1: Status 404 returned error can't find the container with id 5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1
Nov 03 20:03:08.607017 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.606788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:08.607137 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.607072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:08.607137 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.606961    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:08.607222 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.607150    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:09.607134433 +0000 UTC m=+4.091205007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:08.607222 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.607187    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:08.607290 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.607241    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:09.607225148 +0000 UTC m=+4.091295709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:08.707946 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.707828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:08.708090 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.707992    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:08.708090 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.708020    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:08.708090 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.708035    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:08.708260 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:08.708106    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:09.70808634 +0000 UTC m=+4.192156925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:08.945585 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.945456    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:08 +0000 UTC" deadline="2028-04-08 13:59:03.908629053 +0000 UTC"
Nov 03 20:03:08.945585 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.945528    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12521h55m54.963105771s"
Nov 03 20:03:08.982795 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.982753    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerStarted","Data":"9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6"}
Nov 03 20:03:08.986527 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.986489    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zgrz9" event={"ID":"210eccbe-3f54-4885-90d1-c90a4634627a","Type":"ContainerStarted","Data":"49dd36221505f1ca0eed9886e892135dab11e813a2ffdafade3eb1570d77e3d8"}
Nov 03 20:03:08.990966 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.990929    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6"}
Nov 03 20:03:08.994893 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:08.994855    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts" event={"ID":"e49fbaa8-c462-47e9-8cf7-9b20d42180e8","Type":"ContainerStarted","Data":"0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb"}
Nov 03 20:03:09.000687 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.000568    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7e8a73df7412882cc22060fa0198ede" containerID="9916f9f7dbc06766f0a05d04420a22138fd90151d52f7abe767cb2f523781dfb" exitCode=0
Nov 03 20:03:09.000995 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.000913    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerDied","Data":"9916f9f7dbc06766f0a05d04420a22138fd90151d52f7abe767cb2f523781dfb"}
Nov 03 20:03:09.006188 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.005663    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal" event={"ID":"7a01309248349c50ad695700872f4edd","Type":"ContainerStarted","Data":"e4f9ce7dc0abbfeb02277bb709ec2b5f8f15e65297810b8af6926b97d6cecbac"}
Nov 03 20:03:09.010428 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.010350    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4ll8" event={"ID":"d5ef0ef3-bf92-40c9-898c-4e481c4448ee","Type":"ContainerStarted","Data":"5582b04c21474e0f8eaeca1873eed6993fb4794d2444658e40d5962bc1bed392"}
Nov 03 20:03:09.013710 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.013680    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jqq6k" event={"ID":"2f983d9e-13b0-40a1-99a9-8e31927ae731","Type":"ContainerStarted","Data":"5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1"}
Nov 03 20:03:09.019280 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.019222    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4jcr" event={"ID":"4cbb7cff-dc60-4331-834b-416dd7d685a5","Type":"ContainerStarted","Data":"62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8"}
Nov 03 20:03:09.037234 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.037189    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t4x2d" event={"ID":"7d9853ca-6edf-4ccb-b311-e2f57137b882","Type":"ContainerStarted","Data":"7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e"}
Nov 03 20:03:09.047093 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.046833    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nbmrq" event={"ID":"f063b36a-960c-4ed2-bcc5-d92e7268a094","Type":"ContainerStarted","Data":"baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2"}
Nov 03 20:03:09.615288 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.615253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:09.615594 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.615313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:09.615594 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.615424    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:09.615594 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.615470    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:11.615456227 +0000 UTC m=+6.099526787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:09.616004 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.615863    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:09.616004 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.615918    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:11.615901927 +0000 UTC m=+6.099972502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:09.716030 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.715989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:09.716230 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.716214    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:09.716290 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.716237    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:09.716290 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.716252    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:09.716392 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.716320    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:11.716300019 +0000 UTC m=+6.200370600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:09.946183 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.946073    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:08 +0000 UTC" deadline="2028-06-24 21:22:03.888861514 +0000 UTC"
Nov 03 20:03:09.946183 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.946120    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14377h18m53.942747232s"
Nov 03 20:03:09.961005 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.960758    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:09.961005 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.960900    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:09.961391 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.961354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:09.961490 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.961467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:09.963147 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:09.963120    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:09.963268 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:09.963217    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:10.067758 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:10.067705    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerStarted","Data":"bf343dc99437c7204687dbfef4a5b3f16e4218141807af0fd17b2d45625270c6"}
Nov 03 20:03:10.108910 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:10.108821    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal" podStartSLOduration=2.108797843 podStartE2EDuration="2.108797843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:09.080704876 +0000 UTC m=+3.564775465" watchObservedRunningTime="2026-11-03 20:03:10.108797843 +0000 UTC m=+4.592868431"
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.632274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.632352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.632490    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.632555    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:15.63253568 +0000 UTC m=+10.116606269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.633003    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:11.633200 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.633055    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:15.633038122 +0000 UTC m=+10.117108700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:11.733111 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.733047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:11.733428 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.733312    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:11.733428 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.733337    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:11.733428 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.733354    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:11.733428 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.733418    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:15.733396995 +0000 UTC m=+10.217467560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:11.962573 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.961747    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:11.962573 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.961911    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:11.962573 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.962371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:11.962573 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.962476    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:11.963430 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:11.963016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:11.963430 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:11.963118    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:12.084621 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:12.083845    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="e64e698929cb4e16022a960c0cf3212ddf85a7eaac4d6fbb80a7038dd9cec38e" exitCode=0
Nov 03 20:03:12.084621 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:12.083901    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"e64e698929cb4e16022a960c0cf3212ddf85a7eaac4d6fbb80a7038dd9cec38e"}
Nov 03 20:03:12.154667 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:12.154581    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" podStartSLOduration=5.154536881 podStartE2EDuration="5.154536881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:10.109414528 +0000 UTC m=+4.593485115" watchObservedRunningTime="2026-11-03 20:03:12.154536881 +0000 UTC m=+6.638607459"
Nov 03 20:03:13.091640 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:13.091152    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jqq6k" event={"ID":"2f983d9e-13b0-40a1-99a9-8e31927ae731","Type":"ContainerStarted","Data":"d7fe4f377f927631eeafb2c14ad82e7d401807d9442c70e9bc9aaa75a1399f45"}
Nov 03 20:03:13.962578 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:13.962518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:13.962775 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:13.962690    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:13.962775 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:13.962761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:13.962909 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:13.962849    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:13.962909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:13.962889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:13.963020 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:13.962956    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:14.082778 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:14.082697    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:14.083384 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:14.083359    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:14.102063 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:14.101976    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t4x2d" event={"ID":"7d9853ca-6edf-4ccb-b311-e2f57137b882","Type":"ContainerStarted","Data":"f565b4fcd515a0abf61fb5ef9df25c3f17ca2fa5e2140af688ba085cee182270"}
Nov 03 20:03:14.123447 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:14.123376    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jqq6k" podStartSLOduration=3.072727071 podStartE2EDuration="7.123352209s" totalImagesPullingTime="4.050625138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:13.132125473 +0000 UTC m=+7.616196057" watchObservedRunningTime="2026-11-03 20:03:14.123352209 +0000 UTC m=+8.607422793"
Nov 03 20:03:15.114598 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.114544    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts" event={"ID":"e49fbaa8-c462-47e9-8cf7-9b20d42180e8","Type":"ContainerStarted","Data":"83dc0d23b4d113c35ed280c7c59462e9f426b0d9c8e4180645299dfe4dc453d6"}
Nov 03 20:03:15.115110 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.114721    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:15.667361 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.667311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:15.667553 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.667403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:15.667634 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.667553    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:15.667689 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.667642    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:23.667619914 +0000 UTC m=+18.151690491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:15.668123 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.668101    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:15.668207 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.668167    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:23.668149941 +0000 UTC m=+18.152220521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:15.770850 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.770540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:15.771585 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.770782    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:15.771585 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.771378    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:15.771585 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.771393    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:15.771585 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.771461    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:23.771440007 +0000 UTC m=+18.255510579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:15.804059 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.803917    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:03:15.962664 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.962561    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:15.962826 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.962715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:15.962826 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.962739    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:15.962826 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.962798    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:15.963004 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.962848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:15.963004 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:15.962913    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:15.977026 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.976911    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:03:15.803932403Z","UUID":"99db1f69-9806-4122-b31a-67350d16bc82","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:03:15.978997 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.978956    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:03:15.979764 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:15.979317    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:03:16.126630 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:16.126588    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts" event={"ID":"e49fbaa8-c462-47e9-8cf7-9b20d42180e8","Type":"ContainerStarted","Data":"8822843dc276009bd3ff35b7b70374970c163a8385f5de06199003d27d73ac9b"}
Nov 03 20:03:17.131043 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.130939    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4jcr" event={"ID":"4cbb7cff-dc60-4331-834b-416dd7d685a5","Type":"ContainerStarted","Data":"7f01b5ade0c7ab93f0843246e618235cc27a27055fefc6aad356eea8e1fe818a"}
Nov 03 20:03:17.133080 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.132999    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nbmrq" event={"ID":"f063b36a-960c-4ed2-bcc5-d92e7268a094","Type":"ContainerStarted","Data":"daac9e16a406de20244377b472757d6c4ce42e23be6dedb7684c10fd878a104e"}
Nov 03 20:03:17.135580 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.135532    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerStarted","Data":"63aae3eb517387e5d2862f8c0422a62f76e44661b141a28f1ee066bd3cd67cf5"}
Nov 03 20:03:17.175376 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.175314    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t4x2d" podStartSLOduration=4.864520608 podStartE2EDuration="10.175298795s" totalImagesPullingTime="5.310778187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:14.164485036 +0000 UTC m=+8.648555646" watchObservedRunningTime="2026-11-03 20:03:17.175298795 +0000 UTC m=+11.659369376"
Nov 03 20:03:17.218110 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.218048    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c4jcr" podStartSLOduration=1.676690077 podStartE2EDuration="10.218026671s" totalImagesPullingTime="8.541336594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:17.175382301 +0000 UTC m=+11.659452878" watchObservedRunningTime="2026-11-03 20:03:17.218026671 +0000 UTC m=+11.702097254"
Nov 03 20:03:17.218455 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.218221    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nbmrq" podStartSLOduration=2.126634334 podStartE2EDuration="10.218203759s" totalImagesPullingTime="8.091569425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:17.217700344 +0000 UTC m=+11.701770927" watchObservedRunningTime="2026-11-03 20:03:17.218203759 +0000 UTC m=+11.702274341"
Nov 03 20:03:17.959854 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.959609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:17.960027 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.959875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:17.960027 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:17.959988    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:17.960402 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:17.960379    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:17.960450 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:17.960435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:17.960514 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:17.960489    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:18.141482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.140512    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="63aae3eb517387e5d2862f8c0422a62f76e44661b141a28f1ee066bd3cd67cf5" exitCode=0
Nov 03 20:03:18.141482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.140634    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"63aae3eb517387e5d2862f8c0422a62f76e44661b141a28f1ee066bd3cd67cf5"}
Nov 03 20:03:18.143798 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.143758    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zgrz9" event={"ID":"210eccbe-3f54-4885-90d1-c90a4634627a","Type":"ContainerStarted","Data":"6ee9d23df4c0c57e8ba087744ff4e7e246b4df715f7ce890786933f1cc142440"}
Nov 03 20:03:18.147545 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.147509    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts" event={"ID":"e49fbaa8-c462-47e9-8cf7-9b20d42180e8","Type":"ContainerStarted","Data":"bdb868ea21fade065e052c7dab1449cada8f4f7291dabcb4d8eb946f3e7888ce"}
Nov 03 20:03:18.265948 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.265879    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts" podStartSLOduration=3.764601785 podStartE2EDuration="12.265854083s" totalImagesPullingTime="8.501252298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:18.227287465 +0000 UTC m=+12.711358074" watchObservedRunningTime="2026-11-03 20:03:18.265854083 +0000 UTC m=+12.749924666"
Nov 03 20:03:18.266126 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:18.266096    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zgrz9" podStartSLOduration=3.145939788 podStartE2EDuration="11.266085251s" totalImagesPullingTime="8.120145463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:18.265605647 +0000 UTC m=+12.749676231" watchObservedRunningTime="2026-11-03 20:03:18.266085251 +0000 UTC m=+12.750155834"
Nov 03 20:03:19.150939 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.150880    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d4ll8" event={"ID":"d5ef0ef3-bf92-40c9-898c-4e481c4448ee","Type":"ContainerStarted","Data":"355496752c62edb583ef5f878c5f8482b26eefcf0aaaa3a8367cf08a02259268"}
Nov 03 20:03:19.153789 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.153726    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="d118f2d33f03bb969e4e7483cdf4d4f885151924cb9ac41821092ca88c41076a" exitCode=0
Nov 03 20:03:19.153933 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.153857    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"d118f2d33f03bb969e4e7483cdf4d4f885151924cb9ac41821092ca88c41076a"}
Nov 03 20:03:19.188453 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.188313    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d4ll8" podStartSLOduration=1.529920213 podStartE2EDuration="12.18829711s" totalImagesPullingTime="10.658376897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:19.187866543 +0000 UTC m=+13.671937121" watchObservedRunningTime="2026-11-03 20:03:19.18829711 +0000 UTC m=+13.672367688"
Nov 03 20:03:19.960105 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.960072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:19.960285 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.960138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:19.960285 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:19.960138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:19.960285 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:19.960230    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:19.960596 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:19.960567    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:19.960719 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:19.960658    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:21.161188 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:21.161150    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="838f62f4f889843eaefe9f1bf181e8b0bf550ba643519e56b2ef2a2c662889d8" exitCode=0
Nov 03 20:03:21.161890 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:21.161211    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"838f62f4f889843eaefe9f1bf181e8b0bf550ba643519e56b2ef2a2c662889d8"}
Nov 03 20:03:21.960082 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:21.960044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:21.960261 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:21.960050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:21.960261 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:21.960178    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:21.960261 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:21.960236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:21.960463 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:21.960268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:21.960463 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:21.960349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:23.169093 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.168883    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerStarted","Data":"b30a5a21c4c799709e5de91018c50f59ab4e130e95d76aa654ea6d6065ac9d90"}
Nov 03 20:03:23.730581 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.730549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:23.730757 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.730607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:23.730757 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.730696    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:23.730757 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.730727    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:23.730757 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.730753    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:03:39.730738985 +0000 UTC m=+34.214809559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:23.730915 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.730780    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:39.730762148 +0000 UTC m=+34.214832723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:23.831158 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.831079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:23.831280 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.831228    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:23.831280 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.831249    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:23.831280 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.831258    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:23.831394 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.831305    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:39.831290717 +0000 UTC m=+34.315361275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:23.959769 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.959725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:23.959769 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.959741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:23.960015 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.959852    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:23.960015 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.959899    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:23.960015 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:23.959941    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:23.960141 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:23.960027    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:24.174334 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174078    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"1481afa56553cfc4bcbd8da9598e1e132d7146acebef3c50f5faa48802ef1b11"}
Nov 03 20:03:24.174806 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174334    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"45690babc799db113cb0d48ef81d9d4da81bf6673386343f42d2bdd4b1e388e0"}
Nov 03 20:03:24.174806 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174370    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"11d5063937d612e68a03458db9154c297eae21c9710baa51dc39a6e8487a6717"}
Nov 03 20:03:24.174806 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174415    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"74a4f099ca3e660493a07a6a5cc71e101acd22848b36be1b51166b5d6cabe73b"}
Nov 03 20:03:24.174806 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174462    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"efda88c1aca899918b0e4d133d4f411b5f7b9a3f18488e29b7d95127e70b10e0"}
Nov 03 20:03:24.174806 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.174505    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"849a33a643a7630fb37219a0ca960277633a2002a04a554f8deda8e778a9770a"}
Nov 03 20:03:24.177030 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.177001    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="b30a5a21c4c799709e5de91018c50f59ab4e130e95d76aa654ea6d6065ac9d90" exitCode=0
Nov 03 20:03:24.177122 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:24.177041    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"b30a5a21c4c799709e5de91018c50f59ab4e130e95d76aa654ea6d6065ac9d90"}
Nov 03 20:03:25.182371 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.182331    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="9c69fcbbbcf9e5ff8e2c46b899112c995b091a4323a191b3eeb1b7455e416a5e" exitCode=0
Nov 03 20:03:25.182814 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.182410    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerDied","Data":"9c69fcbbbcf9e5ff8e2c46b899112c995b091a4323a191b3eeb1b7455e416a5e"}
Nov 03 20:03:25.614834 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.614794    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:25.614994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.614926    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:25.615397 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.615378    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:03:25.960760 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.960662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:25.960916 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:25.960784    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:25.960916 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.960825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:25.960916 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:25.960853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:25.960916 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:25.960906    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:25.961129 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:25.960963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:26.188121 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:26.188038    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" event={"ID":"b2414e39-9288-425f-9007-319e0815c130","Type":"ContainerStarted","Data":"dca9f4f9bedd5f370e67f46ecd7a3512e0d8d86ac904cdfbbc103f421a2cd9ce"}
Nov 03 20:03:26.190974 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:26.190948    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"f863af7b491b141bcea5919000f1b7cf16e53526feef7e29b393020123fceabe"}
Nov 03 20:03:26.252496 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:26.252440    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h8tqh" podStartSLOduration=9.195875176 podStartE2EDuration="20.252427282s" totalImagesPullingTime="11.056552106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:26.24038786 +0000 UTC m=+20.724458440" watchObservedRunningTime="2026-11-03 20:03:26.252427282 +0000 UTC m=+20.736497860"
Nov 03 20:03:27.959770 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:27.959733    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:27.960435 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:27.959750    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:27.960435 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:27.959835    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:27.960435 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:27.959860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:27.960435 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:27.959954    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:27.960435 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:27.960095    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:28.198982 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.198897    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" event={"ID":"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73","Type":"ContainerStarted","Data":"62e43a5042f8cc1ce074999fba88ad3a7901fb49382c60aaa30d669b5e97511f"}
Nov 03 20:03:28.199130 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.199090    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:28.199191 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.199142    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:28.199242 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.199191    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:28.216487 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.216460    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:28.217987 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.217966    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:03:28.278588 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:28.278507    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd" podStartSLOduration=6.254487094 podStartE2EDuration="21.278492285s" totalImagesPullingTime="15.024005191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:28.278064837 +0000 UTC m=+22.762135417" watchObservedRunningTime="2026-11-03 20:03:28.278492285 +0000 UTC m=+22.762562865"
Nov 03 20:03:29.794392 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.794310    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-th9cw"]
Nov 03 20:03:29.795017 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.794997    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:29.795137 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:29.795115    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:29.796176 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.796150    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vbqn"]
Nov 03 20:03:29.796291 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.796236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:29.796355 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:29.796299    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:29.805904 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.805876    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7mnpc"]
Nov 03 20:03:29.806020 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:29.805970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:29.806071 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:29.806056    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:31.959514 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:31.959481    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:31.959957 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:31.959481    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:31.959957 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:31.959627    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:31.959957 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:31.959492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:31.959957 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:31.959663    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:31.959957 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:31.959758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:33.959368 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:33.959333    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:33.959860 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:33.959339    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:33.959860 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:33.959482    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:33.959860 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:33.959521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:33.959860 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:33.959613    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:33.959860 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:33.959685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:35.960461 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:35.960419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:35.960857 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:35.960505    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:35.960857 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:35.960589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:35.960857 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:35.960677    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:35.960857 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:35.960720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:35.960857 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:35.960776    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:37.084266 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:37.084094    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4jcr_4cbb7cff-dc60-4331-834b-416dd7d685a5/dns-node-resolver/0.log"
Nov 03 20:03:37.960369 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:37.960330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:37.960519 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:37.960391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:37.960519 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:37.960390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:37.960519 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:37.960488    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:37.960655 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:37.960554    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:37.960655 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:37.960603    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:38.230183 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:38.230108    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t4x2d_7d9853ca-6edf-4ccb-b311-e2f57137b882/node-ca/0.log"
Nov 03 20:03:39.735322 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.735290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:39.735713 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.735342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:39.735713 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.735438    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:39.735713 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.735440    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:39.735713 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.735488    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs podName:344332e3-81dd-4a75-a048-b3e35541513d nodeName:}" failed. No retries permitted until 2026-11-03 20:04:11.735475522 +0000 UTC m=+66.219546080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs") pod "network-metrics-daemon-th9cw" (UID: "344332e3-81dd-4a75-a048-b3e35541513d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:39.735713 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.735500    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret podName:e97d10c4-05fb-460d-ba88-87646ce6c054 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:11.735494324 +0000 UTC m=+66.219564881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret") pod "global-pull-secret-syncer-7mnpc" (UID: "e97d10c4-05fb-460d-ba88-87646ce6c054") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:39.836351 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.836258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:39.836514 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.836408    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:39.836514 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.836424    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:39.836514 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.836433    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fwpfl for pod openshift-network-diagnostics/network-check-target-5vbqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:39.836514 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.836483    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl podName:855bb0e3-a462-4d53-a449-06dae0e58db2 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:11.836471315 +0000 UTC m=+66.320541885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwpfl" (UniqueName: "kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl") pod "network-check-target-5vbqn" (UID: "855bb0e3-a462-4d53-a449-06dae0e58db2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:39.959716 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.959680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:39.959901 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.959680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:39.959901 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.959794    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:39.959901 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:39.959815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:39.960014 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.959902    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:39.960014 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:39.959954    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:41.959256 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:41.959216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:41.959256 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:41.959243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:41.959767 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:41.959320    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:41.959767 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:41.959385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:41.959767 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:41.959476    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:41.959767 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:41.959578    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:43.959924 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:43.959883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:43.960370 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:43.960005    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:43.960370 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:43.960013    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:43.960370 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:43.960029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:43.960370 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:43.960129    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:43.960370 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:43.960197    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:45.960382 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:45.960345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:45.960812 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:45.960489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:45.960812 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:45.960508    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:45.960812 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:45.960595    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:45.960812 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:45.960634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:45.960812 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:45.960737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:47.960230 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:47.960192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:47.960230 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:47.960222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:47.960676 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:47.960233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:47.960676 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:47.960311    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:47.960676 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:47.960419    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:47.960676 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:47.960475    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:49.959242 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:49.959201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:49.959242 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:49.959246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:49.959803 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:49.959264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:49.959803 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:49.959349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:49.959803 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:49.959483    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:49.959803 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:49.959603    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:51.959995 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:51.959947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:51.960491 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:51.960060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:51.960491 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:51.960069    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:51.960491 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:51.960166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:51.960491 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:51.960209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:51.960491 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:51.960270    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:53.959586 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:53.959509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:53.959981 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:53.959558    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:53.959981 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:53.959679    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7mnpc" podUID="e97d10c4-05fb-460d-ba88-87646ce6c054"
Nov 03 20:03:53.959981 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:53.959509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:53.959981 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:53.959737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5vbqn" podUID="855bb0e3-a462-4d53-a449-06dae0e58db2"
Nov 03 20:03:53.959981 ip-10-0-136-244 kubenswrapper[2575]: E1103 20:03:53.959832    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-th9cw" podUID="344332e3-81dd-4a75-a048-b3e35541513d"
Nov 03 20:03:54.209308 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.209278    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeReady"
Nov 03 20:03:54.209466 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.209403    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:03:54.267218 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.267185    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-469dq"]
Nov 03 20:03:54.269036 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.269019    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.274232 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.274205    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-blp4x"]
Nov 03 20:03:54.277131 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.277109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.283955 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.283929    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:03:54.284090 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.283965    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:03:54.284141 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.284105    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\""
Nov 03 20:03:54.289102 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.289078    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kn5pc"]
Nov 03 20:03:54.290688 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.290673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.298827 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.298799    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-469dq"]
Nov 03 20:03:54.300512 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.300485    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-blp4x"]
Nov 03 20:03:54.311395 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.311367    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kn5pc"]
Nov 03 20:03:54.335751 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.335751 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.335961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.335961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.335961 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.336090 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.336090 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.335980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9cn\" (UniqueName: \"kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.336090 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.336075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.336198 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.336164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd75t\" (UniqueName: \"kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.385268 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385223    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:03:54.385268 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385228    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385288    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385386    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385391    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385388    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385433    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\""
Nov 03 20:03:54.385504 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385498    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:03:54.385725 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.385596    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\""
Nov 03 20:03:54.437346 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.437346 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd75t\" (UniqueName: \"kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jht5t\" (UniqueName: \"kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.437618 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437639    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9cn\" (UniqueName: \"kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.437909 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.437840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.438123 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.438105    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.438181 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.438155    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.441475 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.441449    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.441596 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.441484    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.456646 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.456612    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd75t\" (UniqueName: \"kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.456799 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.456780    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9cn\" (UniqueName: \"kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.538642 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.538609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jht5t\" (UniqueName: \"kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.538921 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.538668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.541239 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.541213    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.557716 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.557682    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jht5t\" (UniqueName: \"kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.581570 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.581514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:54.589897 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.589871    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:03:54.606669 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.606637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:03:54.899986 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.899915    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-469dq"]
Nov 03 20:03:54.903765 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.903740    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-blp4x"]
Nov 03 20:03:54.909097 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:54.909066    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a8be85_1afd_4233_b228_2ba5b4860e96.slice/crio-f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6 WatchSource:0}: Error finding container f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6: Status 404 returned error can't find the container with id f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6
Nov 03 20:03:54.928031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:54.928004    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kn5pc"]
Nov 03 20:03:54.930568 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:54.930521    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb527645c_8285_4fd1_9154_ae950df3ff3c.slice/crio-d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951 WatchSource:0}: Error finding container d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951: Status 404 returned error can't find the container with id d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951
Nov 03 20:03:55.258551 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.258500    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kn5pc" event={"ID":"b527645c-8285-4fd1-9154-ae950df3ff3c","Type":"ContainerStarted","Data":"d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951"}
Nov 03 20:03:55.259980 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.259955    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blp4x" event={"ID":"71a8be85-1afd-4233-b228-2ba5b4860e96","Type":"ContainerStarted","Data":"babfe6a181a286831bc3034cf24fed13ff0a9e0c7c36bc91a9f6d7280387d108"}
Nov 03 20:03:55.260042 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.260016    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blp4x" event={"ID":"71a8be85-1afd-4233-b228-2ba5b4860e96","Type":"ContainerStarted","Data":"f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6"}
Nov 03 20:03:55.261043 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.261023    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-469dq" event={"ID":"7f95c475-6751-400c-b456-ef13c62c7a45","Type":"ContainerStarted","Data":"f7e1327a79f47c7a4777b1514a79baca9ce62e1df6bf44205184fc58013ac7eb"}
Nov 03 20:03:55.724108 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.724031    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"]
Nov 03 20:03:55.726160 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.726124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.753173 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.753145    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cd5wq\""
Nov 03 20:03:55.753336 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.753244    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:03:55.753393 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.753356    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:03:55.753424 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.753414    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:03:55.753492 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.753481    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:03:55.754378 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.754352    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"]
Nov 03 20:03:55.756076 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.756029    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:03:55.848226 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.848034    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.848226 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.848087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.848226 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.848167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gxp\" (UniqueName: \"kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.848493 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.848282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.948836 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.948801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gxp\" (UniqueName: \"kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.949017 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.948868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.949017 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.948910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.949017 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.948938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.949847 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.949794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.952010 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.951982    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.952243 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.952217    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.961880 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.961856    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:03:55.962168 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.962158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:03:55.962453 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.962442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:03:55.969595 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.969564    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gxp\" (UniqueName: \"kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:55.977185 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977105    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:03:55.977185 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977105    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:03:55.977383 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977179    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:03:55.977383 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977301    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:03:55.977629 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977596    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:03:55.977740 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977651    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:03:55.977740 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:55.977707    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:03:56.040868 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:56.040610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:03:56.267250 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:56.267069    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blp4x" event={"ID":"71a8be85-1afd-4233-b228-2ba5b4860e96","Type":"ContainerStarted","Data":"cbe90ee6dc06fc28db7f86844482c7ce54f6cb16e0bc052a2dd2868cd80173f5"}
Nov 03 20:03:56.351437 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:56.351186    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"]
Nov 03 20:03:56.356682 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:56.356658    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e5787e_ed9a_49db_9f07_9c44621d63d1.slice/crio-75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888 WatchSource:0}: Error finding container 75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888: Status 404 returned error can't find the container with id 75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888
Nov 03 20:03:57.230927 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.230891    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:03:57.233837 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.233811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.255241 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.249766    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:03:57.255241 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.250360    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:03:57.260547 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.260487    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:03:57.260547 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.260487    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:03:57.260788 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.260564    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:03:57.260788 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.260576    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:03:57.260788 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.260594    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dwmnx\""
Nov 03 20:03:57.261026 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.261006    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:03:57.261841 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.261816    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:03:57.274690 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.274653    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x" event={"ID":"e5e5787e-ed9a-49db-9f07-9c44621d63d1","Type":"ContainerStarted","Data":"75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888"}
Nov 03 20:03:57.277389 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.277059    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-469dq" event={"ID":"7f95c475-6751-400c-b456-ef13c62c7a45","Type":"ContainerStarted","Data":"6d1de759adfc65070b261c6755dd67425b863e7114fabf4a506ce1056248dcaf"}
Nov 03 20:03:57.277389 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.277112    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-469dq" event={"ID":"7f95c475-6751-400c-b456-ef13c62c7a45","Type":"ContainerStarted","Data":"a07cf14c8a8ba53345d2204883b59178370f143e09c017c3977c723ac90b5217"}
Nov 03 20:03:57.277389 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.277326    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-469dq"
Nov 03 20:03:57.339554 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.315769    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-469dq" podStartSLOduration=1.334733703 podStartE2EDuration="3.31574794s" totalImagesPullingTime="1.981014237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:57.314773339 +0000 UTC m=+51.798843920" watchObservedRunningTime="2026-11-03 20:03:57.31574794 +0000 UTC m=+51.799818522"
Nov 03 20:03:57.361399 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361399 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361678 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361678 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361678 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361678 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.361678 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.361643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462628 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462628 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462848 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462848 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462848 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462848 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.462848 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.462812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.463864 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.463840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.464459 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.464437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.464686 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.464664    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.464983 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.464962    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.466105 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.466078    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.466297 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.466264    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.482491 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.482457    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.546791 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.546753    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:03:57.851306 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:57.851279    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:03:57.855726 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:03:57.855685    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811c2f89_c8a1_490a_89e3_b1735b4ae271.slice/crio-4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84 WatchSource:0}: Error finding container 4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84: Status 404 returned error can't find the container with id 4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84
Nov 03 20:03:58.281988 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.281953    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df975c8fb-xmwfw" event={"ID":"811c2f89-c8a1-490a-89e3-b1735b4ae271","Type":"ContainerStarted","Data":"4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84"}
Nov 03 20:03:58.283495 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.283468    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kn5pc" event={"ID":"b527645c-8285-4fd1-9154-ae950df3ff3c","Type":"ContainerStarted","Data":"64d79a52e2d8ce40e0850f0303711fd6f102ddf5d43425a1292310c88ced4acc"}
Nov 03 20:03:58.286161 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.286131    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-blp4x" event={"ID":"71a8be85-1afd-4233-b228-2ba5b4860e96","Type":"ContainerStarted","Data":"6c4325f0db1ed6eae7547952375b35f50e98881079784e869bb4f5dd4dd84679"}
Nov 03 20:03:58.288236 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.288208    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x" event={"ID":"e5e5787e-ed9a-49db-9f07-9c44621d63d1","Type":"ContainerStarted","Data":"b3a6f63b67a703c3341c41842722a399badb0af972a0b9740733fa2dadf15666"}
Nov 03 20:03:58.288390 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.288259    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x" event={"ID":"e5e5787e-ed9a-49db-9f07-9c44621d63d1","Type":"ContainerStarted","Data":"1b0bb58692ee8d35fd4747f46f87e73f4a807bee2c64abbd9bbf8cf19f1a2b76"}
Nov 03 20:03:58.322555 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.322461    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kn5pc" podStartSLOduration=1.121128088 podStartE2EDuration="4.322442669s" totalImagesPullingTime="3.201314581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:58.32208457 +0000 UTC m=+52.806155151" watchObservedRunningTime="2026-11-03 20:03:58.322442669 +0000 UTC m=+52.806513227"
Nov 03 20:03:58.360889 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.360827    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-blp4x" podStartSLOduration=2.044362026 podStartE2EDuration="4.360806821s" totalImagesPullingTime="2.316444795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:58.359878972 +0000 UTC m=+52.843949553" watchObservedRunningTime="2026-11-03 20:03:58.360806821 +0000 UTC m=+52.844877401"
Nov 03 20:03:58.399593 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:03:58.399453    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x" podStartSLOduration=1.964669767 podStartE2EDuration="3.399431027s" totalImagesPullingTime="1.43476126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:58.399060744 +0000 UTC m=+52.883131325" watchObservedRunningTime="2026-11-03 20:03:58.399431027 +0000 UTC m=+52.883501608"
Nov 03 20:04:00.216263 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.216232    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:00.332294 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.332264    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h4cg4"]
Nov 03 20:04:00.338203 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.337832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.392361 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.392323    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:04:00.392664 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.392644    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:04:00.392872 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.392852    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:04:00.392872 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.392868    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89ghp\""
Nov 03 20:04:00.491486 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491686 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491686 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491686 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491803 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcw6p\" (UniqueName: \"kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491803 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491803 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491941 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.491996 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.491976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593240 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593240 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcw6p\" (UniqueName: \"kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593342    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593473 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593842 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593667    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593842 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593842 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.593983 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.593893    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.594514 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.594491    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.594772 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.594751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.596166 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.596145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.596645 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.596620    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.612570 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.612520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcw6p\" (UniqueName: \"kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.653674 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:00.653638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:00.663411 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:00.663380    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77075119_7593_44da_9382_75e29ed85ea5.slice/crio-14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767 WatchSource:0}: Error finding container 14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767: Status 404 returned error can't find the container with id 14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767
Nov 03 20:04:01.300958 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:01.300919    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df975c8fb-xmwfw" event={"ID":"811c2f89-c8a1-490a-89e3-b1735b4ae271","Type":"ContainerStarted","Data":"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"}
Nov 03 20:04:01.302478 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:01.302447    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4cg4" event={"ID":"77075119-7593-44da-9382-75e29ed85ea5","Type":"ContainerStarted","Data":"14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767"}
Nov 03 20:04:01.339547 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:01.339476    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7df975c8fb-xmwfw" podStartSLOduration=1.693213101 podStartE2EDuration="4.339461526s" totalImagesPullingTime="2.646248425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:01.339395315 +0000 UTC m=+55.823465894" watchObservedRunningTime="2026-11-03 20:04:01.339461526 +0000 UTC m=+55.823532105"
Nov 03 20:04:02.307205 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:02.307170    2575 generic.go:356] "Generic (PLEG): container finished" podID="77075119-7593-44da-9382-75e29ed85ea5" containerID="40a0a44fdf1337e0401b3f4c1b9f52206779243637ea01f2ca8f53da3adfc691" exitCode=0
Nov 03 20:04:02.307611 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:02.307243    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4cg4" event={"ID":"77075119-7593-44da-9382-75e29ed85ea5","Type":"ContainerDied","Data":"40a0a44fdf1337e0401b3f4c1b9f52206779243637ea01f2ca8f53da3adfc691"}
Nov 03 20:04:03.312486 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:03.312438    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4cg4" event={"ID":"77075119-7593-44da-9382-75e29ed85ea5","Type":"ContainerStarted","Data":"4a341a56e85ccf35bbab53b933a9fec70202c4087d177998b971aa259c5722f6"}
Nov 03 20:04:03.312896 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:03.312569    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h4cg4" event={"ID":"77075119-7593-44da-9382-75e29ed85ea5","Type":"ContainerStarted","Data":"86e0c24b8ca71e0faa929a947d0679e67e53c5c4640ded646b62320c7048ddd8"}
Nov 03 20:04:05.138499 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.138442    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h4cg4" podStartSLOduration=4.236195326 podStartE2EDuration="5.138427096s" totalImagesPullingTime="823.872622ms" totalInitContainerRuntime="78.359148ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.376649255 +0000 UTC m=+57.860719834" watchObservedRunningTime="2026-11-03 20:04:05.138427096 +0000 UTC m=+59.622497676"
Nov 03 20:04:05.139384 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.139365    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"]
Nov 03 20:04:05.141490 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.141475    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.171273 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.171245    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 20:04:05.171452 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.171342    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4wbpf\""
Nov 03 20:04:05.172214 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.172192    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"]
Nov 03 20:04:05.228475 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.228435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.228696 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.228497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fddm\" (UniqueName: \"kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.329163 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.329129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.329343 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.329181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fddm\" (UniqueName: \"kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.331875 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.331848    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.355791 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.355763    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fddm\" (UniqueName: \"kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.451702 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.451601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:05.744902 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:05.744880    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"]
Nov 03 20:04:05.747475 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:05.747435    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4c317d_9756_4fe6_b9c5_cd6468ca8e7f.slice/crio-d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3 WatchSource:0}: Error finding container d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3: Status 404 returned error can't find the container with id d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3
Nov 03 20:04:06.323889 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:06.323847    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r" event={"ID":"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f","Type":"ContainerStarted","Data":"d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3"}
Nov 03 20:04:07.182659 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.182619    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:07.185559 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.185233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.200743 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.200709    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:07.211925 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.211701    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:04:07.211925 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.211701    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:04:07.211925 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.211711    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:04:07.213454 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.213437    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:04:07.213590 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.213571    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:04:07.213994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.213982    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:04:07.225937 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.225913    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:04:07.290779 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.290749    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:07.305450 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305420    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305468    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305478    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305430    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305429    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305498    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xglnv\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305494    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305521    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3apnarc55k39g\""
Nov 03 20:04:07.305657 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.305555    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:04:07.306256 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.306236    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:04:07.329480 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.329362    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r" event={"ID":"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f","Type":"ContainerStarted","Data":"f97fa0d0cad06165bd80ad59cbcab5593d3877db1beee7c0bc1fbc82eb55d571"}
Nov 03 20:04:07.330006 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.329601    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:07.336446 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.336407    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:07.348058 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348235 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348235 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348235 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348235 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348235 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348482 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348606    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.348750 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.349037 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.349037 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.349037 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.348866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.367033 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.366971    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r" podStartSLOduration=1.050901936 podStartE2EDuration="2.366951332s" totalImagesPullingTime="1.316049396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.366306196 +0000 UTC m=+61.850376779" watchObservedRunningTime="2026-11-03 20:04:07.366951332 +0000 UTC m=+61.851021912"
Nov 03 20:04:07.449586 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.449784 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.449784 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.449784 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.449784 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.449784 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.449955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450031 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450381 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450292    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450753 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450655    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.450753 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.450671    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.451328 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.451297    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.452366 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.452327    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.453174 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.453147    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.453900 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.453416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.455201 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.454197    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.455201 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.454712    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.455201 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.454820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.455201 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.454907    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.455781 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.455759    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.456099 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.456070    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.456552 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.456517    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.456698 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.456680    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.456883 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.456862    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.457293 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.457272    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.458140 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.458122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.458422 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.458400    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.469202 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.469176    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.512085 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.512051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:07.547606 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.547574    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:07.547945 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.547886    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:07.553076 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.553054    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:07.826513 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:07.826398    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:07.828552 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:07.828502    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9a2385_1990_48c4_a25c_6a2be45bb591.slice/crio-c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf WatchSource:0}: Error finding container c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf: Status 404 returned error can't find the container with id c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf
Nov 03 20:04:08.333949 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.333919    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf"}
Nov 03 20:04:08.337816 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.337783    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:08.876441 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.876404    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:04:08.878509 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.878486    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.898400 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.897749    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:04:08.899941 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.899907    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:04:08.967590 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967740 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967740 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967632    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967815 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967815 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967888 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967920 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:08.967954 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:08.967931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069348 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069348 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069590 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069630 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069690 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069842 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069902 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.069902 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.069894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.070046 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.070026    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.070994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.070139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.070994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.070301    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.070994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.070404    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.070994 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.070943    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.073200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.072789    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.073200 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.072839    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.091951 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.091920    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.189065 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.188972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:09.340630 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.340591    2575 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736" exitCode=0
Nov 03 20:04:09.341006 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.340692    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"}
Nov 03 20:04:09.482448 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:09.482427    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:04:09.484805 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:09.484778    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode062a8d2_774a_4cf4_a741_bf91149d8d90.slice/crio-fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6 WatchSource:0}: Error finding container fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6: Status 404 returned error can't find the container with id fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6
Nov 03 20:04:10.346411 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:10.346368    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fcdcfbdb-6hb6r" event={"ID":"e062a8d2-774a-4cf4-a741-bf91149d8d90","Type":"ContainerStarted","Data":"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"}
Nov 03 20:04:10.346907 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:10.346426    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fcdcfbdb-6hb6r" event={"ID":"e062a8d2-774a-4cf4-a741-bf91149d8d90","Type":"ContainerStarted","Data":"fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6"}
Nov 03 20:04:10.385373 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:10.385318    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66fcdcfbdb-6hb6r" podStartSLOduration=2.385303861 podStartE2EDuration="2.385303861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:10.384727721 +0000 UTC m=+64.868798302" watchObservedRunningTime="2026-11-03 20:04:10.385303861 +0000 UTC m=+64.869374440"
Nov 03 20:04:11.797241 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.797194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:11.797804 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.797285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:11.810763 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.810730    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:04:11.810763 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.810749    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:04:11.820500 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.820467    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:11.820812 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.820791    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:11.897716 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.897680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:11.899956 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.899926    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:04:11.907243 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.907205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:11.909872 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.909734    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:04:11.911143 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.911121    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:04:11.917328 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.917301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:11.932414 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.932359    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:04:11.944144 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:11.944086    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:12.211099 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.211020    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7mnpc"]
Nov 03 20:04:12.214993 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:12.214961    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97d10c4_05fb_460d_ba88_87646ce6c054.slice/crio-7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9 WatchSource:0}: Error finding container 7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9: Status 404 returned error can't find the container with id 7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9
Nov 03 20:04:12.215762 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.215740    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:04:12.223325 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.223303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:12.242339 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.242314    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-th9cw"]
Nov 03 20:04:12.244374 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:12.244343    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344332e3_81dd_4a75_a048_b3e35541513d.slice/crio-8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa WatchSource:0}: Error finding container 8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa: Status 404 returned error can't find the container with id 8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa
Nov 03 20:04:12.359963 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.359924    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th9cw" event={"ID":"344332e3-81dd-4a75-a048-b3e35541513d","Type":"ContainerStarted","Data":"8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa"}
Nov 03 20:04:12.361517 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.361483    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7mnpc" event={"ID":"e97d10c4-05fb-460d-ba88-87646ce6c054","Type":"ContainerStarted","Data":"7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9"}
Nov 03 20:04:12.518150 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:12.518120    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5vbqn"]
Nov 03 20:04:12.521490 ip-10-0-136-244 kubenswrapper[2575]: W1103 20:04:12.521455    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855bb0e3_a462_4d53_a449_06dae0e58db2.slice/crio-2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1 WatchSource:0}: Error finding container 2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1: Status 404 returned error can't find the container with id 2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1
Nov 03 20:04:13.369385 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:13.369238    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"}
Nov 03 20:04:13.369385 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:13.369317    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"}
Nov 03 20:04:13.371114 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:13.371049    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vbqn" event={"ID":"855bb0e3-a462-4d53-a449-06dae0e58db2","Type":"ContainerStarted","Data":"2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1"}
Nov 03 20:04:14.381623 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:14.380713    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th9cw" event={"ID":"344332e3-81dd-4a75-a048-b3e35541513d","Type":"ContainerStarted","Data":"9b5c781c4423444adcc3f5e24d1e583673ed6e71194f7c9d32652c4b4fbd476b"}
Nov 03 20:04:14.381623 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:14.380829    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-th9cw" event={"ID":"344332e3-81dd-4a75-a048-b3e35541513d","Type":"ContainerStarted","Data":"4dcd6183c0d9559dbd9ff0e2d7b6a1fd8549d6273f83a03ee3dbc3828ac65e54"}
Nov 03 20:04:14.421203 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:14.421133    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-th9cw" podStartSLOduration=67.254347047 podStartE2EDuration="1m8.421114954s" totalImagesPullingTime="1.166767907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:14.420742372 +0000 UTC m=+68.904812953" watchObservedRunningTime="2026-11-03 20:04:14.421114954 +0000 UTC m=+68.905185535"
Nov 03 20:04:15.407561 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:15.406357    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"}
Nov 03 20:04:15.407561 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:15.406445    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"}
Nov 03 20:04:16.414899 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:16.414870    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"}
Nov 03 20:04:16.415311 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:16.414933    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerStarted","Data":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"}
Nov 03 20:04:16.502074 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:16.502002    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.673198589 podStartE2EDuration="9.501981013s" totalImagesPullingTime="6.812354126s" totalInitContainerRuntime="16.428298ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:16.489245448 +0000 UTC m=+70.973316035" watchObservedRunningTime="2026-11-03 20:04:16.501981013 +0000 UTC m=+70.986051594"
Nov 03 20:04:17.419732 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:17.419698    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5vbqn" event={"ID":"855bb0e3-a462-4d53-a449-06dae0e58db2","Type":"ContainerStarted","Data":"43b3c21ab95db694063218e2797cf832b06250f80102939f10e3dd9b7c624340"}
Nov 03 20:04:17.420201 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:17.419895    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:17.457088 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:17.457044    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5vbqn" podStartSLOduration=67.627578204 podStartE2EDuration="1m11.457027347s" totalImagesPullingTime="3.829449143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.456413882 +0000 UTC m=+71.940484463" watchObservedRunningTime="2026-11-03 20:04:17.457027347 +0000 UTC m=+71.941097930"
Nov 03 20:04:17.512420 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:17.512390    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:17.711232 ip-10-0-136-244 kubenswrapper[2575]: I1103 20:04:17.711123    2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:04:17.711314 ip-10-0-136-244 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:04:17.726059 ip-10-0-136-244 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:04:17.726234 ip-10-0-136-244 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:04:17.726289 ip-10-0-136-244 systemd[1]: kubelet.service: Consumed 4.261s CPU time.
Nov 03 20:04:17.746302 ip-10-0-136-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:04:17.822752 ip-10-0-136-244 kubenswrapper[5424]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:17.822752 ip-10-0-136-244 kubenswrapper[5424]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:04:17.822752 ip-10-0-136-244 kubenswrapper[5424]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:17.822752 ip-10-0-136-244 kubenswrapper[5424]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:17.824918 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824823    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.824918 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824916    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.824918 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824920    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.824918 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824923    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824926    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824930    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824932    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824935    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824937    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824940    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824943    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824946    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824948    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824951    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824953    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824956    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824959    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824962    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824964    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824966    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824969    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824971    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824974    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.825046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824976    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824979    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824982    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824990    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824993    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.824997    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825000    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825002    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825005    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825008    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825010    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825013    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825016    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825019    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825022    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825024    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825027    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825029    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825032    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825035    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.825562 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825038    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825040    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825043    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825052    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825056    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825058    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825061    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825064    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825068    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825073    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825076    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825079    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825082    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825084    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825087    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825090    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825093    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825096    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825098    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.826082 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825100    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825104    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825106    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825109    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825111    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825114    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825117    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825119    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825122    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825125    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825127    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825130    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825132    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825135    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825137    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825140    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825142    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825149    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825152    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.826579 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825154    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825157    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825159    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825162    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825164    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825167    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825169    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825172    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825174    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825177    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825179    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825182    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825184    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825187    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825190    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825193    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825196    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825200    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825203    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.827052 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825206    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825208    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825211    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825215    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825217    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.825220    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826121    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826130    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826133    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826136    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826139    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826142    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826148    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826151    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826154    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826157    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826160    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826164    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826167    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.827603 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826170    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826173    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826175    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826178    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826180    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826183    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826186    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826192    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826194    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826197    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826199    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826202    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826205    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826207    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826210    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826213    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826216    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826218    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826222    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826224    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.828078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826226    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826229    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826232    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826234    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826237    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826240    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826248    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826251    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826254    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826256    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826259    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826261    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826264    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826267    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826269    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826271    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826274    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826276    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826279    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.828622 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826283    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826285    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826288    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826290    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826293    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826296    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826298    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826300    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826303    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826306    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826308    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826311    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826313    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826315    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826318    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826320    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826323    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826325    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826328    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826330    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.829117 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826340    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826343    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826346    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826348    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826351    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826353    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826355    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826358    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826360    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826363    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826365    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826368    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826371    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826374    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826377    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826379    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826382    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826384    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826389    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.829656 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826393    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826396    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826399    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826402    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826404    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826407    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826409    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826412    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826414    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826417    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826419    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826422    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826425    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826427    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826435    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826983    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826991    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826995    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.826998    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.830179 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827000    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827003    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827006    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827009    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827011    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827014    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827018    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827021    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827024    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827027    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827029    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827032    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827035    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827040    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827043    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827045    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827048    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827050    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827053    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827056    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.830652 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827059    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827061    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827064    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827067    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827069    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827072    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827074    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827078    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827080    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827083    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827085    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827088    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827090    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827094    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827098    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827101    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827104    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827107    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827111    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827114    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.831185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827116    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827119    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827121    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827124    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827126    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827130    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827133    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827135    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827138    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827140    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827143    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827145    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827148    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827150    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827153    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827155    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827158    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827160    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827163    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.831715 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827165    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827171    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827174    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827177    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827179    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827182    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827185    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827187    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827190    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827192    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827195    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827197    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827200    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827202    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827205    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827207    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827209    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827212    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827216    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827218    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.832492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827221    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827223    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827226    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827229    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827231    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827234    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827236    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827239    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827242    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827244    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827246    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827249    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827252    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827254    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827257    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827260    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827262    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827264    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827267    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827269    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.833204 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827272    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827274    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.827278    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827377    5424 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827385    5424 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827394    5424 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827401    5424 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827408    5424 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827412    5424 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827419    5424 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827426    5424 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827429    5424 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827433    5424 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827436    5424 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827440    5424 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827443    5424 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827446    5424 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827451    5424 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827454    5424 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827457    5424 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827460    5424 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827464    5424 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827466    5424 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:04:17.833725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827469    5424 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827472    5424 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827476    5424 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827480    5424 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827484    5424 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827488    5424 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827491    5424 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827495    5424 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827498    5424 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827501    5424 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827504    5424 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827507    5424 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827511    5424 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827514    5424 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827517    5424 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827520    5424 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827523    5424 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827539    5424 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827545    5424 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827550    5424 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827553    5424 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827556    5424 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827559    5424 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827563    5424 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:04:17.834309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827567    5424 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827570    5424 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827573    5424 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827576    5424 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827579    5424 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827582    5424 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827585    5424 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827588    5424 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827591    5424 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827594    5424 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827599    5424 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827602    5424 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827606    5424 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827611    5424 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827615    5424 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827618    5424 flags.go:64] FLAG: --help="false"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827621    5424 flags.go:64] FLAG: --hostname-override="ip-10-0-136-244.ec2.internal"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827625    5424 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827628    5424 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827631    5424 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827634    5424 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827637    5424 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827640    5424 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:04:17.834915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827643    5424 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827646    5424 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827649    5424 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827652    5424 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827656    5424 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827659    5424 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827662    5424 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827665    5424 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827668    5424 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827671    5424 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827674    5424 flags.go:64] FLAG: --lock-file=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827677    5424 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827680    5424 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827683    5424 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827689    5424 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827692    5424 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827695    5424 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827698    5424 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827700    5424 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827704    5424 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827707    5424 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827709    5424 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827716    5424 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827720    5424 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827724    5424 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:04:17.835651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827727    5424 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827730    5424 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827733    5424 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827737    5424 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827740    5424 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827743    5424 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827746    5424 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827753    5424 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827757    5424 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827759    5424 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827762    5424 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827767    5424 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827769    5424 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827773    5424 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827775    5424 flags.go:64] FLAG: --port="10250"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827779    5424 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827781    5424 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-089ab518f32319645"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827785    5424 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827787    5424 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827790    5424 flags.go:64] FLAG: --register-node="true"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827793    5424 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827801    5424 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827804    5424 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827806    5424 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827809    5424 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827814    5424 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:04:17.836256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827817    5424 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827820    5424 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827823    5424 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827827    5424 flags.go:64] FLAG: --runonce="false"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827830    5424 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827834    5424 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827836    5424 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827839    5424 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827842    5424 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827846    5424 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827849    5424 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827852    5424 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827855    5424 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827858    5424 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827860    5424 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827863    5424 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827866    5424 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827870    5424 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827873    5424 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827878    5424 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827881    5424 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827884    5424 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827888    5424 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827891    5424 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827894    5424 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:04:17.836899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827897    5424 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827900    5424 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827902    5424 flags.go:64] FLAG: --v="2"
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827907    5424 flags.go:64] FLAG: --version="false"
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827911    5424 flags.go:64] FLAG: --vmodule=""
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827915    5424 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.827918    5424 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828057    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828060    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828063    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828069    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828074    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828078    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828083    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828086    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828089    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828092    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828095    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828097    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828100    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.837661 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828103    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828105    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828108    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828110    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828114    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828117    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828120    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828122    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828125    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828127    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828130    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828133    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828135    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828138    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828140    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828143    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828145    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828148    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828150    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.838166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828153    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828156    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828158    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828162    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828165    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828168    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828170    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828173    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828175    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828178    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828181    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828183    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828185    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828188    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828190    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828193    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828195    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828199    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828201    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828204    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.838651 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828206    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828209    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828211    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828214    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828216    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828219    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828221    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828224    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828226    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828228    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828231    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828233    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828236    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828238    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828241    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828245    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828247    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828250    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828252    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.839385 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828255    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828258    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828261    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828264    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828266    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828269    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828271    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828274    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828276    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828279    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828282    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828285    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828288    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828290    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828293    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828295    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828298    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828300    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828303    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.840104 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828305    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828308    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828311    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828313    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828316    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828318    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828321    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828323    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828325    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828329    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828332    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828334    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828337    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828340    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828342    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828346    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.840698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.828373    5424 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828410    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828414    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828417    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828420    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828423    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828426    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828430    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828433    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828436    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828438    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828441    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828444    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828446    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828449    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828451    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828454    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828456    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828459    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828462    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.841154 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828464    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828467    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828469    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828472    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828474    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828477    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828481    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828484    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828486    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828489    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828491    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828494    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828497    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828500    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828502    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828505    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828507    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828510    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828512    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828516    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.841812 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828519    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828522    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828537    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828540    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828543    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828546    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828548    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828551    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828553    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828556    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828558    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828561    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828564    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828566    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828569    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828571    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828574    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828577    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828581    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828583    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.842492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828586    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828588    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828591    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828594    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828597    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828600    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828603    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828606    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828608    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828610    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828613    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828617    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828619    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828622    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828624    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828627    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828630    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828633    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828635    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.843230 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828638    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828640    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828643    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828646    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828649    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828651    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828654    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828656    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828659    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828661    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828664    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828668    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828671    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828673    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828676    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828679    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828682    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828684    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828687    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828690    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.843958 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828693    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828695    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828698    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828702    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828707    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828710    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828713    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.828717    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.834301    5424 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.834325    5424 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.834352    5424 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.834357    5424 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834423    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834429    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834433    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834436    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834439    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834442    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834445    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.844826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834447    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834450    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834452    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834455    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834457    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834460    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834462    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834465    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834468    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834470    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834473    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834475    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834478    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834481    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834484    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834487    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834489    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834492    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834494    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834497    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.845646 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834499    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834503    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834505    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834508    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834510    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834513    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834516    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834519    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834523    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834543    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834547    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834549    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834552    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834554    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834557    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834560    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834562    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834565    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834567    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.846507 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834570    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834572    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834575    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834578    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834580    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834583    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834586    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834588    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834591    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834594    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834597    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834599    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834602    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834605    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834608    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834611    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834613    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834616    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834619    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.847355 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834621    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834624    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834627    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834629    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834632    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834634    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834636    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834639    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834641    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834644    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834646    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834649    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834652    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834654    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834657    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834660    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834662    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834665    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834668    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834671    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.848175 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834674    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834676    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834679    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834681    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834684    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834686    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834689    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834693    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834696    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834698    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834701    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834703    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834706    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834709    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834711    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834714    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834716    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834719    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834721    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834726    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.849050 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834729    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.834762    5424 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834941    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834946    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834950    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834952    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834955    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834958    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834961    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834963    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834966    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834968    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834971    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834973    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834976    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834978    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.849938 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834981    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834983    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834986    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834989    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834991    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834994    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834997    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.834999    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835002    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835005    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835007    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835010    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835014    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835018    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835021    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835023    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835026    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835028    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835031    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.850626 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835033    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835036    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835039    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835041    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835044    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835048    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835050    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835053    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835055    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835058    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835061    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835063    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835066    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835068    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835071    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835073    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835076    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835078    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835081    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.851402 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835083    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835085    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835088    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835090    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835093    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835095    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835099    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835102    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835106    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835108    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835111    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835114    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835117    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835120    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835123    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835125    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835128    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835130    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835133    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.852199 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835136    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835138    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835141    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835143    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835146    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835149    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835151    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835153    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835156    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835158    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835161    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835163    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835166    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835168    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835171    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835173    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835176    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835178    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835181    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835183    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.852970 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835186    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835188    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835190    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835193    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835196    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835198    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835201    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835204    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835207    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835209    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835212    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835214    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835217    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835223    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835225    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.835249    5424 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:17.853885 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835282    5424 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835286    5424 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835289    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835292    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835295    5424 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835297    5424 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835300    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835302    5424 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835305    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835308    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835311    5424 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835313    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835316    5424 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835318    5424 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835321    5424 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835324    5424 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835326    5424 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835329    5424 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835331    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835334    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:17.854280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835336    5424 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835339    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835342    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835344    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835347    5424 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835349    5424 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835352    5424 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835354    5424 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835357    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835360    5424 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835364    5424 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835366    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835369    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835371    5424 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835373    5424 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835376    5424 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835378    5424 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835381    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835383    5424 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835386    5424 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:17.854797 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835389    5424 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835391    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835395    5424 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835399    5424 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835402    5424 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835405    5424 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835408    5424 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835410    5424 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835413    5424 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835415    5424 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835418    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835420    5424 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835423    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835426    5424 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835428    5424 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835430    5424 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835433    5424 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835436    5424 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835438    5424 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:17.855289 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835440    5424 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835443    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835447    5424 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835449    5424 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835452    5424 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835455    5424 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835457    5424 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835460    5424 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835463    5424 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835466    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835469    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835471    5424 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835473    5424 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835476    5424 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835478    5424 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835482    5424 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835486    5424 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835488    5424 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835491    5424 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:17.855770 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835493    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835496    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835498    5424 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835500    5424 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835503    5424 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835505    5424 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835508    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835510    5424 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835513    5424 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835515    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835517    5424 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835520    5424 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835542    5424 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835546    5424 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835548    5424 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835552    5424 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835555    5424 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835558    5424 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835562    5424 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835565    5424 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:17.856234 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835567    5424 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835570    5424 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835573    5424 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835577    5424 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835580    5424 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835583    5424 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835585    5424 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:17.835588    5424 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.835658    5424 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.836694    5424 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.836767    5424 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.836963    5424 server.go:1014] "Starting client certificate rotation"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.837128    5424 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.837205    5424 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:58:06 +0000 UTC" deadline="2028-05-06 18:25:28.171891952 +0000 UTC"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.837248    5424 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13198h21m10.334647614s"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.837495    5424 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:04:17.856823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.838647    5424 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:04:17.857223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.840087    5424 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:04:17.857223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.842075    5424 log.go:25] "Validated CRI v1 image API"
Nov 03 20:04:17.857223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.842912    5424 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:04:17.857223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.846189    5424 fs.go:135] Filesystem UUIDs: map[73a26d9b-d44c-406a-b5fd-bff4812995a6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d113a559-edde-4a98-a7a8-a4c307836dde:/dev/nvme0n1p4]
Nov 03 20:04:17.857351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.846218    5424 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49dd36221505f1ca0eed9886e892135dab11e813a2ffdafade3eb1570d77e3d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49dd36221505f1ca0eed9886e892135dab11e813a2ffdafade3eb1570d77e3d8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7e1327a79f47c7a4777b1514a79baca9ce62e1df6bf44205184fc58013ac7eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7e1327a79f47c7a4777b1514a79baca9ce62e1df6bf44205184fc58013ac7eb/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/config-reloader/2 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/prometheus/15 major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~empty-dir/config-out major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/kube-api-access-hsbzh:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/kube-api-access-hsbzh major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/tls-assets major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/web-config major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/210eccbe-3f54-4885-90d1-c90a4634627a/volumes/kubernetes.io~projected/kube-api-access-xqn2f:{mountpoint:/var/lib/kubelet/pods/210eccbe-3f54-4885-90d1-c90a4634627a/volumes/kubernetes.io~projected/kube-api-access-xqn2f major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~projected/kube-api-access-7xxmr:{mountpoint:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~projected/kube-api-access-7xxmr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f983d9e-13b0-40a1-99a9-8e31927ae731/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2f983d9e-13b0-40a1-99a9-8e31927ae731/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~projected/kube-api-access-xv2lt:{mountpoint:/var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~projected/kube-api-access-xv2lt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~secret/metrics-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cbb7cff-dc60-4331-834b-416dd7d685a5/volumes/kubernetes.io~projected/kube-api-access-ktb47:{mountpoint:/var/lib/kubelet/pods/4cbb7cff-dc60-4331-834b-416dd7d685a5/volumes/kubernetes.io~projected/kube-api-access-ktb47 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~projected/kube-api-access-nd75t:{mountpoint:/var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~projected/kube-api-access-nd75t major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~projected/kube-api-access-pcw6p:{mountpoint:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~projected/kube-api-access-pcw6p major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d9853ca-6edf-4ccb-b311-e2f57137b882/volumes/kubernetes.io~projected/kube-api-access-lkmng:{mountpoint:/var/lib/kubelet/pods/7d9853ca-6edf-4ccb-b311-e2f57137b882/volumes/kubernetes.io~projected/kube-api-access-lkmng major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~projected/kube-api-access-pm9cn:{mountpoint:/var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~projected/kube-api-access-pm9cn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~projected/kube-api-access-hbb5f:{mountpoint:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~projected/kube-api-access-hbb5f major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/855bb0e3-a462-4d53-a449-06dae0e58db2/volumes/kubernetes.io~projected/kube-api-access-fwpfl:{mountpoint:/var/lib/kubelet/pods/855bb0e3-a462-4d53-a449-06dae0e58db2/volumes/kubernetes.io~projected/kube-api-access-fwpfl major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2414e39-9288-425f-9007-319e0815c130/volumes/kubernetes.io~projected/kube-api-access-5qjwr:{mountpoint:/var/lib/kubelet/pods/b2414e39-9288-425f-9007-319e0815c130/volumes/kubernetes.io~projected/kube-api-access-5qjwr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~projected/kube-api-access-jht5t:{mountpoint:/var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~projected/kube-api-access-jht5t major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~projected/kube-api-access-4bght:{mountpoint:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~projected/kube-api-access-4bght major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~projected/kube-api-access-d5vp7:{mountpoint:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~projected/kube-api-access-d5vp7 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49fbaa8-c462-47e9-8cf7-9b20d42180e8/volumes/kubernetes.io~projected/kube-api-access-2t66m:{mountpoint:/var/lib/kubelet/pods/e49fbaa8-c462-47e9-8cf7-9b20d42180e8/volumes/kubernetes.io~projected/kube-api-access-2t66m major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~projected/kube-api-access-69gxp:{mountpoint:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~projected/kube-api-access-69gxp major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e97d10c4-05fb-460d-ba88-87646ce6c054/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e97d10c4-05fb-460d-ba88-87646ce6c054/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~projected/kube-api-access-6fddm:{mountpoint:/var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~projected/kube-api-access-6fddm major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f063b36a-960c-4ed2-bcc5-d92e7268a094/volumes/kubernetes.io~projected/kube-api-access-6qn5r:{mountpoint:/var/lib/kubelet/pods/f063b36a-960c-4ed2-bcc5-d92e7268a094/volumes/kubernetes.io~projected/kube-api-access-6qn5r major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fe62e9d48b60b0b592e0e094a520a92cb2b14b21df973adc5781a6d2060d2cea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e6a1f155edbb9457d6bb949fad01184bdc8410533c38cf68dcaf06a8775deaf7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/30aff0614f576779ed3c3e7670dc233b846bb716d3ca827908fbc364a5cad0bf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f0fd5f31c1a777aad15ee6a468054e5096e8057fb1d2b568f320c7f4b9294d77/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a177dd22bbc3b69d947be14c21273755b1f49500f8b469274c183daad344e09f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/41a5424a235ad452a32592f88f0a74c4e62ddb33da3202fa6c44b77724f47382/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dcfde98db27cbc77c009e65955caf0395432630349e2105d186dfa9c784bcced/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0dedea6a01bd013a2b36df6e93537bcee9663f78706500e159bbbbbd159583b3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/26eecad8cff0f1e85c07d2aaee104b9fc12d12724c5328e4f3d9114d547e6513/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1e7a0c1b5f489c79543d098f60b6a89294c36a1ddcc9136347b366ac4c2093dd/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ab968857fd6384d638317c41e68fc057020055eeff984836fb213d1230bd66ff/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/f2c0e9bae391b72b3a75ef45e2a2515262449a3a845e1a48e1fca3702f48b61c/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/86a1b62907ebb879f155876d37c6838b1b35d1e96b31f52c5a18f4ca082602a9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7e78df6ab8e4b7a40b676fc2c42096d6d9970006e9c67fd37e993ff58117102f/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6acf14670608986abd837f2c4956c9969590cd14d4956e3bf9f690677647569e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0c6d6efe2881e453e754514695ea5e2bbade66ac49115f9941c4e0c980d9ebaf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6cfc25186a22ab59e94c78423f08b647f54129eb6c99d6f6e8fef94de00997ab/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/220b24a6e6de8c38a63c464c4a9a21e76546ed16ad66cdd58908af399742e330/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/70140099553a7820bd59bf8c7d8e6f9d366305c3d40111bad3ddfbc1470a65df/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6c077b80bba73d030a724d9577abb6407c357741b49efb84a9da44bb58f1ce02/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/00af35546f79e75c5a409cb88255e507c6234497e32e63e339b1100a2ece7154/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/436efee9663e5bb9709b3b722953f7e73e0020890d6829fd05b466305c80fef7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ad7e2210bb44c95b6976b75dc5a29ba40c70627ead757380edbce146bb805343/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/19e037f4d15383e64612b330d3bc99d916d9ff151d3721bb861422abd5d0e7c2/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0554ddd9cf6df4eb0a6c88196faa4a8bbaec1916afff027bab1233c8c80dc655/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/a2799743744cd3c090f764eb4c8913ac1f262123aed756377d378dda94a0b2a3/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/653d5f935626d0b6245a380d96882ce082932b3f396a6fcd6da18a0f31bc6289/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/f507ecf7bf4248a78285ebe1eb52c1fb6f6159da3d753069e169ea59fcac92e7/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/a46d20fbaa68eff78a3796373aa8240cac3fc31bfe6ed47ffb4cd38807454698/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/3ad2c48c9b9646a278faad6f9e2280f00a352766c5a1c7e516e3ce71d49449d5/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/3fe11d8fdfe1db0d7efeef27aac22c91daa234469fa937ec70c09575fe80fc38/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/de4eb85de86af1cf61a6dd214312408c005e804e1b0387f61e805105e15518d3/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ced4732fea218355c30ce8902fd88cf27ce9005285c20acddc71bfb5c890d390/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0c3a0f737c071b7264fdfb10874f43aaff0fa7b3f92e5c9f84b90b25103afc9d/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/917c2698c6900c5c56cd193e3986b6ef163010a61efd91f174d0ab1cb9e6dc85/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2219865f5cb4ae8cb097cbf95e04b8c5c8aa59968afefabd7737781ec4b5c618/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/e1bdb5969b34314b440eb0def08e3b533af583f42d36c31770b91111dca26dae/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/c9d030e4cb015f5be7a4946610119d8110362fb918c3299da6d464fb4509a27c/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/f874e63d2f751c2e07cd542ee73310a1d5124ee5aa45f1cf50d6872d41cd3ca0/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/5228c01a662886f85726c66a0527f9eaa9247f8582264f626be996aee18f4e77/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/b01c24d733e1d975dc2a4e44434ea960881876270e2f194344db073ded44bc3d/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/7b222192ce76d9d2097b1a7e0902d1dab35a770584c0b414197a288be8fc530b/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/e449f4ff28de2ae69773c1e57e7ae5b44e070c949833d78e7547bae32c9e37da/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/b4e56ef5c69ef43ff7a8f47537d1d1bb2cdd3537724ef7cb63876c8d579c0809/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/ae0641ba3a5b99967e727412506073afa9fbd07cab7d693e76f4873e681363d3/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/99e8af009d35b2e2cafbf8a35307da0537f0620fc7a84ef567daa5e2e5f29a1c/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/52c2e1785935155bf51385ab712c3a15598aef6adbd74be3fd0bd8b4f203882f/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/eaf4987b1f640f8aaef830c6a396a67d2c3534e5420a53d9776140e5a32ede35/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/debf953271a599fd19f79f1712d828364c560045f97434da56dd28baaf19d97d/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/043295bf256cc67ffc2236ddac8ebe6f1335f0464c952241bd84ebafd3701ffc/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/5ebd010016253f109319b449f59b976fac6938b522b82ff12764cd3f0ba82c64/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/b6cea9f1a7eb0b78646bb39db8b3e61f04bbe51e50d2279c9491fe79d76531f8/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/ba4640768d7d710d4f538dea4ac75020a8306ec30619a851e81046838470bd8a/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/bca3c974e7ce1057e8f5d70643c8af3ddc7c900eec82af8c13e259d2a7e82fb1/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/2a000911eaa734bb8cca38399ed5951d746b4114de16a02d6d9abc681ba8894f/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/eb0b6024aaaba3e0cf61d07b0b442c8bd429fd013d0f6659b0a09fcac7a5c040/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e0268a3d7443b47d8aadd578a2972a0d512fe806412d21066d4f8de1df982c69/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6dc22f45cd4d8097784de5aa6ff0cdd84a5306af7b540db99b602155701ebb91/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7fe5a4dec1c6544fce5a6c122be1f651a319ec9cdfbfd69fda02580ce4ac4300/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/fcafee342996f6531c300fb25d2e54f4b0d0fe2b10fd28018e931ba3f2ac46f1/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e6ccc5d7eb4cae72e97dcb6b10cb7269ee97dcec94bb41b780ba36d2721e626a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8806855bcb7bf29e23dc780d2de5ae334882cc96e5dde006c83e44874470a6cd/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fa638434acb878279ee2dcc1a78de218fd91dcab0ec202a0338c59d3c2d3699a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e64733db9825c47d4076cbdfb719bc6bcbe066c3567bbfa57512b8a41ec48efa/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3d1517d9159c262e3be43725e40fcfacc5c12245edc82fae197c181aa2c43775/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7f756087d66c0cc95a59ab0dcabd54f6e1831bba406f0907d3fdf0a9f7cf10ee/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 20:04:17.858153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.857705    5424 manager.go:217] Machine: {Timestamp:2026-11-03 20:04:17.85719505 +0000 UTC m=+0.066211039 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097857 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2808e94028c5ba20b667f4c8a14ad3 SystemUUID:ec2808e9-4028-c5ba-20b6-67f4c8a14ad3 BootID:2967ee86-3264-4b87-a319-b32ad7f4420f Filesystems:[{Device:/var/lib/kubelet/pods/f063b36a-960c-4ed2-bcc5-d92e7268a094/volumes/kubernetes.io~projected/kube-api-access-6qn5r DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baaa4e9c925daec395b0ba204d770955186eda6d3dc94916b2c08985f9a504d2/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volumes/kubernetes.io~projected/kube-api-access-7xxmr DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~projected/kube-api-access-69gxp DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b2414e39-9288-425f-9007-319e0815c130/volumes/kubernetes.io~projected/kube-api-access-5qjwr DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e49fbaa8-c462-47e9-8cf7-9b20d42180e8/volumes/kubernetes.io~projected/kube-api-access-2t66m DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62ba62534ecb2e1ccc900c0eb2ec663c2c03c725f4bd141b70b45cfa6e4399b8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2598f1e98ce6d9c38b90ea3149a41c19eefd9e7e9ee06ce5f6d7ccfd659a7ff1/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~projected/kube-api-access-xv2lt DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d63d719f2f8fc34f50c1ad2d1bf8e5810c0fcda61bf21d24c69ccd8d02a4d951/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~projected/kube-api-access-hsbzh DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4cbb7cff-dc60-4331-834b-416dd7d685a5/volumes/kubernetes.io~projected/kube-api-access-ktb47 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7e1327a79f47c7a4777b1514a79baca9ce62e1df6bf44205184fc58013ac7eb/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~projected/kube-api-access-hbb5f DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:319 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75e1d6db5e8ffcf69ef315bfc209b9970708da299c51161ae1cc286e61cf6888/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/344332e3-81dd-4a75-a048-b3e35541513d/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~projected/kube-api-access-pm9cn DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ee24aedd45317f88dfa92a1866e69f4143e89e33731bf3d97236b40228aa58e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e4e5dfd473b6b23baf7cde0f8fa5997d6542fc72869d47f5ac86cf584fd11e6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f95c475-6751-400c-b456-ef13c62c7a45/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d5ef0ef3-bf92-40c9-898c-4e481c4448ee/volumes/kubernetes.io~projected/kube-api-access-4bght DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:248 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/210eccbe-3f54-4885-90d1-c90a4634627a/volumes/kubernetes.io~projected/kube-api-access-xqn2f DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3f746f71a3b8584a43a5ff4d570802cf555e0a123f52eed0e96a66932c424c6/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/2f983d9e-13b0-40a1-99a9-8e31927ae731/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/volumes/kubernetes.io~projected/kube-api-access-6fddm DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2f6a8bd94d8cb4dd1c74cdc9e9f70918598a14ab1f2f64943e0a052c17e73c3/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e5787e-ed9a-49db-9f07-9c44621d63d1/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/77075119-7593-44da-9382-75e29ed85ea5/volumes/kubernetes.io~projected/kube-api-access-pcw6p DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14d1e2194567d647bf28488c5b7febd2a8bf07e34b43f32b05f6fc81f312d767/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:315 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d9853ca-6edf-4ccb-b311-e2f57137b882/volumes/kubernetes.io~projected/kube-api-access-lkmng DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49dd36221505f1ca0eed9886e892135dab11e813a2ffdafade3eb1570d77e3d8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes/kubernetes.io~projected/kube-api-access-d5vp7 DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b4b6551e3856e748f799adfa08f0dd08b464e1b9c97a0adaae17c62677402b9/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8038343c7674bc29ccbe56b724008f1f581dc1ec7f5bdea39314eda4074f04fa/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/855bb0e3-a462-4d53-a449-06dae0e58db2/volumes/kubernetes.io~projected/kube-api-access-fwpfl DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d7de159e5e93fd64dcf9f2ab53531ae286008311cad4fd85535492e2236fffb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b527645c-8285-4fd1-9154-ae950df3ff3c/volumes/kubernetes.io~projected/kube-api-access-jht5t DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b7ba370e15ddfa89f3e78b4e24e0ad6643bb4eb29324e4c39298eeec3e9c1a1/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71a8be85-1afd-4233-b228-2ba5b4860e96/volumes/kubernetes.io~projected/kube-api-access-nd75t DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:320 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e97d10c4-05fb-460d-ba88-87646ce6c054/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d1092776c03a0bbbc9e2a80d52ef49dc80f623ae01355d9759243d2fdb253e6/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2598f1e98ce6d9c MacAddress:b2:c2:7e:ba:f2:f9 Speed:10000 Mtu:8901} {Name:4f3b78fdef642d2 MacAddress:ba:86:ca:ac:e7:c8 Speed:10000 Mtu:8901} {Name:75e1d6db5e8ffcf MacAddress:7a:72:42:2c:42:d8 Speed:10000 Mtu:8901} {Name:7b4b6551e3856e7 MacAddress:d2:75:76:5b:78:62 Speed:10000 Mtu:8901} {Name:8038343c7674bc2 MacAddress:1a:fe:63:20:cc:22 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:f7:b8:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:6b:87:ad:88:ba Speed:0 Mtu:8901} {Name:c91309eeefce474 MacAddress:da:f7:56:0c:e9:07 Speed:10000 Mtu:8901} {Name:d2f6a8bd94d8cb4 MacAddress:52:78:c1:ff:ab:99 Speed:10000 Mtu:8901} {Name:d63d719f2f8fc34 MacAddress:2a:7c:96:63:9c:e9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:f7:b8:21 Speed:0 Mtu:9001} {Name:f3f746f71a3b858 MacAddress:1a:8c:ad:67:c7:22 Speed:10000 Mtu:8901} {Name:f7e1327a79f47c7 MacAddress:c6:4a:fa:62:7f:14 Speed:10000 Mtu:8901} {Name:fc8eded88344ae5 MacAddress:0e:93:c8:9b:0f:3f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:46:5d:79:89:85:52 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:37:cb:a2:9d:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:04:17.858379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858158    5424 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:04:17.858379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858212    5424 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:04:17.858696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858661    5424 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:04:17.858867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858695    5424 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:04:17.858955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858872    5424 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:04:17.858955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858885    5424 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:04:17.858955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858900    5424 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:04:17.858955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.858930    5424 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:04:17.859097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859087    5424 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:04:17.859367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859351    5424 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:04:17.859498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859486    5424 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:04:17.859586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859556    5424 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:04:17.859586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859573    5424 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:04:17.859586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859586    5424 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:04:17.859712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859600    5424 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:04:17.859712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.859611    5424 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:04:17.860485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.860467    5424 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:04:17.860485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.860485    5424 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:04:17.861157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.861139    5424 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:04:17.864219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864203    5424 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:04:17.864219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864217    5424 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:04:17.864323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864318    5424 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:04:17.864548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864518    5424 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:04:17.864582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864568    5424 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:04:17.864687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864678    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:04:17.864719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864693    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:04:17.864719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864700    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:04:17.864719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864705    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:04:17.864719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864711    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:04:17.864719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864717    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864723    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864729    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864736    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864742    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864754    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864770    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864829    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:04:17.864854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.864834    5424 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:04:17.868540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.868512    5424 server.go:1263] "Started kubelet"
Nov 03 20:04:17.868626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.868598    5424 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:04:17.869050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.868842    5424 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:04:17.869050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.868992    5424 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:04:17.869236 ip-10-0-136-244 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:04:17.869418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.869327    5424 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:04:17.871355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.871330    5424 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.873962    5424 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.873998    5424 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874137    5424 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:08 +0000 UTC" deadline="2028-05-19 10:54:46.874434358 +0000 UTC"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874162    5424 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13502h50m29.00027547s"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:17.874194    5424 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-244.ec2.internal\" not found"
Nov 03 20:04:17.874717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874713    5424 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:04:17.875071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874895    5424 factory.go:55] Registering systemd factory
Nov 03 20:04:17.875071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874917    5424 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:04:17.875291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.874111    5424 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:04:17.875344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875295    5424 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:04:17.875344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875315    5424 factory.go:153] Registering CRI-O factory
Nov 03 20:04:17.875444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875353    5424 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:04:17.875444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875418    5424 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:04:17.875444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875441    5424 factory.go:103] Registering Raw factory
Nov 03 20:04:17.875681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875458    5424 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:04:17.875811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.875796    5424 manager.go:319] Starting recovery of all containers
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878553    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878602    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878619    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73" volumeName="kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878633    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73" volumeName="kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878645    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f983d9e-13b0-40a1-99a9-8e31927ae731" volumeName="kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878660    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="344332e3-81dd-4a75-a048-b3e35541513d" volumeName="kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878671    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e5787e-ed9a-49db-9f07-9c44621d63d1" volumeName="kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:17.878721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878703    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e5787e-ed9a-49db-9f07-9c44621d63d1" volumeName="kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878746    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878760    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2414e39-9288-425f-9007-319e0815c130" volumeName="kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878773    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b527645c-8285-4fd1-9154-ae950df3ff3c" volumeName="kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878785    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878796    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878826    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f" volumeName="kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878858    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878870    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="210eccbe-3f54-4885-90d1-c90a4634627a" volumeName="kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878882    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73" volumeName="kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878896    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878943    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878955    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878968    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d9853ca-6edf-4ccb-b311-e2f57137b882" volumeName="kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.878992    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879006    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879021    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879034    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879047    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="210eccbe-3f54-4885-90d1-c90a4634627a" volumeName="kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879058    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879070    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879089    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e97d10c4-05fb-460d-ba88-87646ce6c054" volumeName="kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879118    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879133    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879146    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cbb7cff-dc60-4331-834b-416dd7d685a5" volumeName="kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879158    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879169    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879181    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f" volumeName="kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879195    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879207    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="344332e3-81dd-4a75-a048-b3e35541513d" volumeName="kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879233    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a8be85-1afd-4233-b228-2ba5b4860e96" volumeName="kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879245    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879257    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d9853ca-6edf-4ccb-b311-e2f57137b882" volumeName="kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879268    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f95c475-6751-400c-b456-ef13c62c7a45" volumeName="kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879282    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2414e39-9288-425f-9007-319e0815c130" volumeName="kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879294    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e5787e-ed9a-49db-9f07-9c44621d63d1" volumeName="kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879306    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879318    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ef0ef3-bf92-40c9-898c-4e481c4448ee" volumeName="kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879331    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ef0ef3-bf92-40c9-898c-4e481c4448ee" volumeName="kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879342    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879354    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f063b36a-960c-4ed2-bcc5-d92e7268a094" volumeName="kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879365    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879376    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f95c475-6751-400c-b456-ef13c62c7a45" volumeName="kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879398    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5ef0ef3-bf92-40c9-898c-4e481c4448ee" volumeName="kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879410    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879431    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73" volumeName="kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879443    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73" volumeName="kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879454    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879465    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879486    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2414e39-9288-425f-9007-319e0815c130" volumeName="kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879498    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879509    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49fbaa8-c462-47e9-8cf7-9b20d42180e8" volumeName="kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879523    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879563    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a8be85-1afd-4233-b228-2ba5b4860e96" volumeName="kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879575    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879591    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879604    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879616    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e5787e-ed9a-49db-9f07-9c44621d63d1" volumeName="kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879632    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="811c2f89-c8a1-490a-89e3-b1735b4ae271" volumeName="kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879644    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f983d9e-13b0-40a1-99a9-8e31927ae731" volumeName="kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879655    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cbb7cff-dc60-4331-834b-416dd7d685a5" volumeName="kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879667    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a8be85-1afd-4233-b228-2ba5b4860e96" volumeName="kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879678    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879689    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f95c475-6751-400c-b456-ef13c62c7a45" volumeName="kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879701    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879714    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879726    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879737    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a8be85-1afd-4233-b228-2ba5b4860e96" volumeName="kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879749    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="855bb0e3-a462-4d53-a449-06dae0e58db2" volumeName="kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879759    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2414e39-9288-425f-9007-319e0815c130" volumeName="kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879772    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879784    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e062a8d2-774a-4cf4-a741-bf91149d8d90" volumeName="kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879797    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f063b36a-960c-4ed2-bcc5-d92e7268a094" volumeName="kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879809    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9a2385-1990-48c4-a25c-6a2be45bb591" volumeName="kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879821    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77075119-7593-44da-9382-75e29ed85ea5" volumeName="kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879833    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f95c475-6751-400c-b456-ef13c62c7a45" volumeName="kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879846    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b527645c-8285-4fd1-9154-ae950df3ff3c" volumeName="kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879858    5424 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f063b36a-960c-4ed2-bcc5-d92e7268a094" volumeName="kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r" seLinuxMountContext=""
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879869    5424 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:04:17.884555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.879908    5424 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:04:17.910089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.910041    5424 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:04:17.911743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.911716    5424 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:04:17.911743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.911743    5424 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:04:17.911919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.911766    5424 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:04:17.911919 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:17.911807    5424 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:04:17.922990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.922962    5424 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:17.923137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.923089    5424 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:04:17.923805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.923783    5424 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:17.937119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.937083    5424 generic.go:356] "Generic (PLEG): container finished" podID="b7e8a73df7412882cc22060fa0198ede" containerID="9916f9f7dbc06766f0a05d04420a22138fd90151d52f7abe767cb2f523781dfb" exitCode=0
Nov 03 20:04:17.948945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.948866    5424 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:17.949242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.949215    5424 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:17.950737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.950711    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736" exitCode=0
Nov 03 20:04:17.958329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958295    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="9c69fcbbbcf9e5ff8e2c46b899112c995b091a4323a191b3eeb1b7455e416a5e" exitCode=0
Nov 03 20:04:17.958329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958326    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="b30a5a21c4c799709e5de91018c50f59ab4e130e95d76aa654ea6d6065ac9d90" exitCode=0
Nov 03 20:04:17.958567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958338    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="838f62f4f889843eaefe9f1bf181e8b0bf550ba643519e56b2ef2a2c662889d8" exitCode=0
Nov 03 20:04:17.958567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958344    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="d118f2d33f03bb969e4e7483cdf4d4f885151924cb9ac41821092ca88c41076a" exitCode=0
Nov 03 20:04:17.958567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958352    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="63aae3eb517387e5d2862f8c0422a62f76e44661b141a28f1ee066bd3cd67cf5" exitCode=0
Nov 03 20:04:17.958567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.958357    5424 generic.go:356] "Generic (PLEG): container finished" podID="b2414e39-9288-425f-9007-319e0815c130" containerID="e64e698929cb4e16022a960c0cf3212ddf85a7eaac4d6fbb80a7038dd9cec38e" exitCode=0
Nov 03 20:04:17.964968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.964894    5424 generic.go:356] "Generic (PLEG): container finished" podID="77075119-7593-44da-9382-75e29ed85ea5" containerID="40a0a44fdf1337e0401b3f4c1b9f52206779243637ea01f2ca8f53da3adfc691" exitCode=0
Nov 03 20:04:17.982159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:17.982133    5424 manager.go:324] Recovery completed
Nov 03 20:04:18.006273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006237    5424 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:04:18.006273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006263    5424 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:04:18.006470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006290    5424 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:04:18.006501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006468    5424 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:04:18.006501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006482    5424 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:04:18.006582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006505    5424 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:04:18.006582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006514    5424 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:04:18.006582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006522    5424 policy_none.go:50] "Start"
Nov 03 20:04:18.006582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006559    5424 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:04:18.006582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.006573    5424 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:04:18.007578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.007565    5424 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:04:18.007619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.007578    5424 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:04:18.007619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.007583    5424 policy_none.go:44] "Start"
Nov 03 20:04:18.012017 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:18.011999    5424 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 20:04:18.012100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.012075    5424 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:04:18.014697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.014686    5424 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:04:18.015174 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:18.014709    5424 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:04:18.015241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015177    5424 server.go:87] "Starting device plugin registration server"
Nov 03 20:04:18.015449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015440    5424 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:04:18.015484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015451    5424 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:04:18.015599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015575    5424 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:04:18.015727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015690    5424 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:04:18.015793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015737    5424 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:04:18.015793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015745    5424 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:04:18.015883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.015796    5424 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:04:18.015725726Z","UUID":"74d58ef6-a7c0-4b9b-9746-ea9b4dc91f4f","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:04:18.017291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.017261    5424 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:04:18.017370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.017296    5424 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:04:18.017416 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:18.017394    5424 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:04:18.116237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.116183    5424 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:04:18.118323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.118304    5424 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:04:18.118508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.118334    5424 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:04:18.118508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.118380    5424 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-244.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:04:18.118508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.118425    5424 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.153149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.153118    5424 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.153270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.153211    5424 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.212802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.212745    5424 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"]
Nov 03 20:04:18.213442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213426    5424 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.213516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213428    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerStarted","Data":"bf343dc99437c7204687dbfef4a5b3f16e4218141807af0fd17b2d45625270c6"}
Nov 03 20:04:18.213516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213493    5424 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.213516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213497    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerDied","Data":"9916f9f7dbc06766f0a05d04420a22138fd90151d52f7abe767cb2f523781dfb"}
Nov 03 20:04:18.213516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213513    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal" event={"ID":"b7e8a73df7412882cc22060fa0198ede","Type":"ContainerStarted","Data":"1ea9563d307d0f19538fb7a4a58acd56c27cea89635fbbcb554baba7fa7f29bf"}
Nov 03 20:04:18.213698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213575    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal" event={"ID":"7a01309248349c50ad695700872f4edd","Type":"ContainerStarted","Data":"e4f9ce7dc0abbfeb02277bb709ec2b5f8f15e65297810b8af6926b97d6cecbac"}
Nov 03 20:04:18.213698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.213601    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal" event={"ID":"7a01309248349c50ad695700872f4edd","Type":"ContainerStarted","Data":"9244dbddd1a9ac6ebde35798eaf15393a46f29d58e22ca0ce2bc4dd9c60fdfd1"}
Nov 03 20:04:18.240513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.240480    5424 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:18.240604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.240500    5424 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:18.240604 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:18.240582    5424 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.240672 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:18.240621    5424 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.323993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.323960    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.323993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.323993    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.324144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.324011    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424232    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424365    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424306    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424391    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424435    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a01309248349c50ad695700872f4edd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" (UID: \"7a01309248349c50ad695700872f4edd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.424452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.424437    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7e8a73df7412882cc22060fa0198ede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" (UID: \"b7e8a73df7412882cc22060fa0198ede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.860091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.860043    5424 apiserver.go:52] "Watching apiserver"
Nov 03 20:04:18.915142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.915105    5424 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:04:18.915740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.915713    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-nbmrq","openshift-ingress-canary/ingress-canary-kn5pc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal","kube-system/global-pull-secret-syncer-7mnpc","kube-system/konnectivity-agent-jqq6k","openshift-console/console-7df975c8fb-xmwfw","openshift-network-diagnostics/network-check-target-5vbqn","openshift-network-operator/iptables-alerter-zgrz9","openshift-dns/dns-default-469dq","openshift-image-registry/node-ca-t4x2d","openshift-insights/insights-runtime-extractor-blp4x","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts","openshift-monitoring/node-exporter-h4cg4","kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal","openshift-console/console-66fcdcfbdb-6hb6r","openshift-ovn-kubernetes/ovnkube-node-vz4pd","openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r","openshift-multus/network-metrics-daemon-th9cw","openshift-cluster-node-tuning-operator/tuned-d4ll8","openshift-dns/node-resolver-c4jcr","openshift-multus/multus-additional-cni-plugins-h8tqh"]
Nov 03 20:04:18.931758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.931725    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:04:18.932703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932672    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:04:18.932827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932759    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\""
Nov 03 20:04:18.932827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932765    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\""
Nov 03 20:04:18.932827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932798    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\""
Nov 03 20:04:18.932827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932804    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\""
Nov 03 20:04:18.932827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932818    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:04:18.933066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932943    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:04:18.933066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932972    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:04:18.933066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.932982    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:04:18.933066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933021    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:04:18.933066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933059    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933069    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933129    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933136    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933140    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933145    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933166    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933184    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933188    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933214    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.933274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.933217    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.942512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.942494    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.942646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.942632    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\""
Nov 03 20:04:18.943730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.943716    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.944066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.944050    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.944140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.944106    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:04:18.944272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.944259    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:04:18.945060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945000    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945074    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945103    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945165    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945179    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945193    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.945265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945208    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\""
Nov 03 20:04:18.945484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945336    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.945572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945552    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.945673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945664    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:04:18.945723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945677    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.945723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945698    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:04:18.945723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945705    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:04:18.945723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945717    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:04:18.945892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945725    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.945892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945720    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:04:18.945892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945757    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:04:18.945892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.945789    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.946239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.946226    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:04:18.947572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.947555    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:04:18.947696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.947686    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:04:18.948625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948588    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.948625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948621    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:04:18.948793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948775    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:04:18.948952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948903    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:04:18.948952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948930    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:04:18.948952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948941    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.949335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.948918    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:04:18.949335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949020    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:04:18.949335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949231    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 20:04:18.949335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949287    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949462    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949470    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949499    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949505    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949517    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4wbpf\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949549    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949565    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949591    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949639    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949664    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xglnv\""
Nov 03 20:04:18.949836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949773    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:04:18.950299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949876    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\""
Nov 03 20:04:18.950299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.949944    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:04:18.950299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.950219    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:04:18.950299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.950238    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:04:18.955639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.955601    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:04:18.955753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.955644    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:04:18.955753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.955702    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:04:18.955753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.955720    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:04:18.956675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.956659    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.957697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.957683    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cd5wq\""
Nov 03 20:04:18.957781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.957714    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.957781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.957722    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:04:18.958141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958125    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:04:18.958196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958143    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 20:04:18.958196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958167    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 20:04:18.958196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958183    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3apnarc55k39g\""
Nov 03 20:04:18.958324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958180    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:04:18.958324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958289    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:04:18.958324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958324    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dwmnx\""
Nov 03 20:04:18.958906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.958894    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89ghp\""
Nov 03 20:04:18.962903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.962882    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:04:18.963683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.963666    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:04:18.963888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.963875    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:04:18.976646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.976610    5424 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:04:18.989773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.989739    5424 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:18.989958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:18.989941    5424 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:19.013523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.013497    5424 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:19.013690 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:19.013568    5424 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-244.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:19.013992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.013977    5424 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:19.014050 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:19.014022    5424 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-244.ec2.internal"
Nov 03 20:04:19.027220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027190    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.027220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027224    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.027402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027242    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jht5t\" (UniqueName: \"kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:04:19.027402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027259    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.027402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027315    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.027402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027352    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:19.027402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027380    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027408    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027435    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027461    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027487    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027512    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027599    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.027649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027609    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f95c475-6751-400c-b456-ef13c62c7a45-metrics-tls\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027686    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027693    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027712    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027762    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9cn\" (UniqueName: \"kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027766    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027794    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027823    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027833    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027852    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027866    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e5787e-ed9a-49db-9f07-9c44621d63d1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.027957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027890    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f983d9e-13b0-40a1-99a9-8e31927ae731-agent-certs\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028000    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028029    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxmr\" (UniqueName: \"kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.027998    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028065    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028083    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028162    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028188    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028242    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71a8be85-1afd-4233-b228-2ba5b4860e96-data-volume\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028279    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028303    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028411    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028438    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028460    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.028507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028474    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028508    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028551    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028562    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-config\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028573    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028602    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028631    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028657    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028721    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qn5r\" (UniqueName: \"kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028731    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028753    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028789    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028794    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-cni-binary-copy\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028817    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028849    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028864    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovn-node-metrics-cert\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028878    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t66m\" (UniqueName: \"kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.028980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028958    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.028958    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2lt\" (UniqueName: \"kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029010    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029023    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029057    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029090    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029096    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029117    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029142    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029192    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029218    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029237    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029254    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.029346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029347    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029391    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029422    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029437    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029454    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029496    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029513    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-tmp\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029595    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029620    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d9853ca-6edf-4ccb-b311-e2f57137b882-serviceca\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029659    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029662    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029690    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029692    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029721    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029722    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-tls\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029722    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029746    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029788    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.029875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029802    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029911    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029927    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/210eccbe-3f54-4885-90d1-c90a4634627a-iptables-alerter-script\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029925    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029960    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-metrics-client-ca\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029966    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.029991    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030004    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-ovnkube-script-lib\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030003    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030043    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030060    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030096    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb47\" (UniqueName: \"kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030117    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030156    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn2f\" (UniqueName: \"kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030175    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030185    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030237    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcw6p\" (UniqueName: \"kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030272    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030285    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030298    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030335    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030361    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030472    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gxp\" (UniqueName: \"kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030504    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030544    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030567    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030588    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030650    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030678    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030686    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e97d10c4-05fb-460d-ba88-87646ce6c054-original-pull-secret\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030713    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030740    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030739    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.030758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030771    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fddm\" (UniqueName: \"kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030790    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030820    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030844    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.030948    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031023    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031049    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031063    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031082    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031106    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031169    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031187    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031213    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031234    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-env-overrides\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031242    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031253    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95c475-6751-400c-b456-ef13c62c7a45-config-volume\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031270    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bght\" (UniqueName: \"kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031296    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031315    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbb7cff-dc60-4331-834b-416dd7d685a5-tmp-dir\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031347    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031498    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031557    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e5787e-ed9a-49db-9f07-9c44621d63d1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031572    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd75t\" (UniqueName: \"kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031590    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031605    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031629    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031659    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031681    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f95c475-6751-400c-b456-ef13c62c7a45-tmp-dir\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031684    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031723    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031750    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031768    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031804    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031849    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031894    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b527645c-8285-4fd1-9154-ae950df3ff3c-cert\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031918    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031944    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031978    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.031997    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032012    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/344332e3-81dd-4a75-a048-b3e35541513d-metrics-certs\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032024    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjwr\" (UniqueName: \"kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032043    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032061    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032079    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.032078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032094    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032139    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-tuned\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032154    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032180    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032197    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032266    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032288    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032302    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032327    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032332    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032357    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-daemon-config\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032360    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032384    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032398    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032437    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032474    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032502    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032582    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032607    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71a8be85-1afd-4233-b228-2ba5b4860e96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032612    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032646    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032655    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2414e39-9288-425f-9007-319e0815c130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032650    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032714    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032743    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032764    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032810    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032833    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032836    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032853    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032856    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032890    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032911    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032926    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032947    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032959    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.032987    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033065    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033075    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f983d9e-13b0-40a1-99a9-8e31927ae731-konnectivity-ca\") pod \"konnectivity-agent-jqq6k\" (UID: \"2f983d9e-13b0-40a1-99a9-8e31927ae731\") " pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033093    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033114    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033140    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033177    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-textfile\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033223    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmng\" (UniqueName: \"kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.033692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.033248    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.046439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046405    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jht5t\" (UniqueName: \"kubernetes.io/projected/b527645c-8285-4fd1-9154-ae950df3ff3c-kube-api-access-jht5t\") pod \"ingress-canary-kn5pc\" (UID: \"b527645c-8285-4fd1-9154-ae950df3ff3c\") " pod="openshift-ingress-canary/ingress-canary-kn5pc"
Nov 03 20:04:19.046650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046412    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9cn\" (UniqueName: \"kubernetes.io/projected/7f95c475-6751-400c-b456-ef13c62c7a45-kube-api-access-pm9cn\") pod \"dns-default-469dq\" (UID: \"7f95c475-6751-400c-b456-ef13c62c7a45\") " pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:19.046650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046459    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxmr\" (UniqueName: \"kubernetes.io/projected/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-kube-api-access-7xxmr\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.046650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046588    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"console-66fcdcfbdb-6hb6r\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") " pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:19.046650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046648    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qn5r\" (UniqueName: \"kubernetes.io/projected/f063b36a-960c-4ed2-bcc5-d92e7268a094-kube-api-access-6qn5r\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.046944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.046925    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2lt\" (UniqueName: \"kubernetes.io/projected/344332e3-81dd-4a75-a048-b3e35541513d-kube-api-access-xv2lt\") pod \"network-metrics-daemon-th9cw\" (UID: \"344332e3-81dd-4a75-a048-b3e35541513d\") " pod="openshift-multus/network-metrics-daemon-th9cw"
Nov 03 20:04:19.047156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.047138    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t66m\" (UniqueName: \"kubernetes.io/projected/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kube-api-access-2t66m\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.047701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.047681    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb47\" (UniqueName: \"kubernetes.io/projected/4cbb7cff-dc60-4331-834b-416dd7d685a5-kube-api-access-ktb47\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.048053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.048034    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpfl\" (UniqueName: \"kubernetes.io/projected/855bb0e3-a462-4d53-a449-06dae0e58db2-kube-api-access-fwpfl\") pod \"network-check-target-5vbqn\" (UID: \"855bb0e3-a462-4d53-a449-06dae0e58db2\") " pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:19.049250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.049226    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gxp\" (UniqueName: \"kubernetes.io/projected/e5e5787e-ed9a-49db-9f07-9c44621d63d1-kube-api-access-69gxp\") pod \"prometheus-operator-9cd89b74c-hqr7x\" (UID: \"e5e5787e-ed9a-49db-9f07-9c44621d63d1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hqr7x"
Nov 03 20:04:19.049356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.049343    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fddm\" (UniqueName: \"kubernetes.io/projected/eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f-kube-api-access-6fddm\") pod \"monitoring-plugin-77f4f76594-hcz9r\" (UID: \"eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:19.049422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.049403    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcw6p\" (UniqueName: \"kubernetes.io/projected/77075119-7593-44da-9382-75e29ed85ea5-kube-api-access-pcw6p\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.049646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.049630    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd75t\" (UniqueName: \"kubernetes.io/projected/71a8be85-1afd-4233-b228-2ba5b4860e96-kube-api-access-nd75t\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.050327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050308    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn2f\" (UniqueName: \"kubernetes.io/projected/210eccbe-3f54-4885-90d1-c90a4634627a-kube-api-access-xqn2f\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.050416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050332    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bght\" (UniqueName: \"kubernetes.io/projected/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-kube-api-access-4bght\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.050416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050364    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"prometheus-k8s-0\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:19.050416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050374    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"console-7df975c8fb-xmwfw\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") " pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:19.050517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050447    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjwr\" (UniqueName: \"kubernetes.io/projected/b2414e39-9288-425f-9007-319e0815c130-kube-api-access-5qjwr\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.050761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.050744    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmng\" (UniqueName: \"kubernetes.io/projected/7d9853ca-6edf-4ccb-b311-e2f57137b882-kube-api-access-lkmng\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133631    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133678    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133710    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133735    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133776    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133801    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133826    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133850    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133873    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133896    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133900    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-kubelet\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133922    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134000    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-ovn\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133997    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-dbus\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133917    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-etc-kubernetes\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134062    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-sys\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134074    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-sys\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134079    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-multus\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134130    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-netd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133996    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-node-exporter-wtmp\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.133860    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-kubelet\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134202    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-cnibin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134211    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134243    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134246    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-system-cni-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134280    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134293    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-systemd\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.134274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134305    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134331    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77075119-7593-44da-9382-75e29ed85ea5-root\") pod \"node-exporter-h4cg4\" (UID: \"77075119-7593-44da-9382-75e29ed85ea5\") " pod="openshift-monitoring/node-exporter-h4cg4"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134331    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134367    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134389    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134379    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134438    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-device-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134438    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-multus-certs\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134460    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134502    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134551    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134568    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134575    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134592    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-k8s-cni-cncf-io\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134620    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134622    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4cbb7cff-dc60-4331-834b-416dd7d685a5-hosts-file\") pod \"node-resolver-c4jcr\" (UID: \"4cbb7cff-dc60-4331-834b-416dd7d685a5\") " pod="openshift-dns/node-resolver-c4jcr"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134666    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134690    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134688    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134714    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134709    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-var-lib-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134666    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-var-lib-kubelet\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134748    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-conf-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134763    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-log-socket\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134750    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134797    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-run-netns\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134794    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134829    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134835    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e97d10c4-05fb-460d-ba88-87646ce6c054-kubelet-config\") pod \"global-pull-secret-syncer-7mnpc\" (UID: \"e97d10c4-05fb-460d-ba88-87646ce6c054\") " pod="kube-system/global-pull-secret-syncer-7mnpc"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134853    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134878    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134910    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134917    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d9853ca-6edf-4ccb-b311-e2f57137b882-host\") pod \"node-ca-t4x2d\" (UID: \"7d9853ca-6edf-4ccb-b311-e2f57137b882\") " pod="openshift-image-registry/node-ca-t4x2d"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134960    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-netns\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134941    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134998    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135034    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-system-cni-dir\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135072    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-node-log\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135073    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135025    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-run-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135113    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysconfig\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135119    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.134802    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-sys-fs\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135145    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135167    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-host-var-lib-cni-bin\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135168    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135200    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135212    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135226    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135243    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-systemd\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135249    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135279    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135282    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135303    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135325    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135349    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135373    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135396    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135417    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135616    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135646    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135661    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135677    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135692    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135708    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135725    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135741    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135764    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135790    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135858    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-run-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135889    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210eccbe-3f54-4885-90d1-c90a4634627a-host-slash\") pod \"iptables-alerter-zgrz9\" (UID: \"210eccbe-3f54-4885-90d1-c90a4634627a\") " pod="openshift-network-operator/iptables-alerter-zgrz9"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135457    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2414e39-9288-425f-9007-319e0815c130-cnibin\") pod \"multus-additional-cni-plugins-h8tqh\" (UID: \"b2414e39-9288-425f-9007-319e0815c130\") " pod="openshift-multus/multus-additional-cni-plugins-h8tqh"
Nov 03 20:04:19.135873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135915    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-slash\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135950    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-run\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.135974    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-kubernetes\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136017    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-systemd-units\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136027    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-lib-modules\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136054    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-hostroot\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136065    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-host\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136077    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-etc-openvswitch\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136113    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71a8be85-1afd-4233-b228-2ba5b4860e96-crio-socket\") pod \"insights-runtime-extractor-blp4x\" (UID: \"71a8be85-1afd-4233-b228-2ba5b4860e96\") " pod="openshift-insights/insights-runtime-extractor-blp4x"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136117    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f063b36a-960c-4ed2-bcc5-d92e7268a094-multus-socket-dir-parent\") pod \"multus-nbmrq\" (UID: \"f063b36a-960c-4ed2-bcc5-d92e7268a094\") " pod="openshift-multus/multus-nbmrq"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136162    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-registration-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136200    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-sysctl-conf\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136215    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e49fbaa8-c462-47e9-8cf7-9b20d42180e8-socket-dir\") pod \"aws-ebs-csi-driver-node-drtts\" (UID: \"e49fbaa8-c462-47e9-8cf7-9b20d42180e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-drtts"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136255    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ef0ef3-bf92-40c9-898c-4e481c4448ee-etc-modprobe-d\") pod \"tuned-d4ll8\" (UID: \"d5ef0ef3-bf92-40c9-898c-4e481c4448ee\") " pod="openshift-cluster-node-tuning-operator/tuned-d4ll8"
Nov 03 20:04:19.138363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.136265    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73-host-cni-bin\") pod \"ovnkube-node-vz4pd\" (UID: \"2a3830e1-66fd-4ea7-a6ba-1800e3fbaf73\") " pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:19.186370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.186341    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:04:19.926714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:19.926684    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-469dq"
Nov 03 20:04:20.049590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:20.049546    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:20.054240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:20.054217    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:20.293228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:20.293196    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:21.647217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:21.647191    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:21.735694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:21.735659    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hcz9r"
Nov 03 20:04:22.307822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:22.307786    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:22.323940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:22.323908    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:22.420085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:22.420046    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:22.747371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:22.747314    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:27.502645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:27.502568    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:27.503026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:27.502800    5424 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:04:31.251859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:31.251831    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vz4pd"
Nov 03 20:04:32.424948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:32.424918    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:04:41.031762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:41.031722    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:42.293434 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:42.293388    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:04:45.018268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.018219    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7df975c8fb-xmwfw" podUID="811c2f89-c8a1-490a-89e3-b1735b4ae271" containerName="console" containerID="cri-o://6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c" gracePeriod=15
Nov 03 20:04:45.261938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.261910    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:45.337431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337340    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337389    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337412    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337431    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337451    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337568    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337595    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") pod \"811c2f89-c8a1-490a-89e3-b1735b4ae271\" (UID: \"811c2f89-c8a1-490a-89e3-b1735b4ae271\") "
Nov 03 20:04:45.337779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337749    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:45.337976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.337864    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:45.338110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.338084    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:45.338216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.338194    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:45.340250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.340224    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:45.340334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.340272    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "kube-api-access-hbb5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:45.340334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.340277    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert" pod "811c2f89-c8a1-490a-89e3-b1735b4ae271" (UID: "811c2f89-c8a1-490a-89e3-b1735b4ae271"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:45.438843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438808    5424 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-oauth-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.438843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438842    5424 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/811c2f89-c8a1-490a-89e3-b1735b4ae271-tmp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.439063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438856    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbb5f\" (UniqueName: \"kubernetes.io/projected/811c2f89-c8a1-490a-89e3-b1735b4ae271-kube-api-access-hbb5f\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.439063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438870    5424 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.439063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438882    5424 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-oauth-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.439063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438893    5424 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.439063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.438905    5424 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/811c2f89-c8a1-490a-89e3-b1735b4ae271-console-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:45.865863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:45.865821    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jqq6k"
Nov 03 20:04:46.079587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.079557    5424 generic.go:356] "Generic (PLEG): container finished" podID="811c2f89-c8a1-490a-89e3-b1735b4ae271" containerID="6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c" exitCode=0
Nov 03 20:04:46.080039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.079648    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df975c8fb-xmwfw" event={"ID":"811c2f89-c8a1-490a-89e3-b1735b4ae271","Type":"ContainerDied","Data":"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"}
Nov 03 20:04:46.080039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.079661    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7df975c8fb-xmwfw"
Nov 03 20:04:46.080039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.079684    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7df975c8fb-xmwfw" event={"ID":"811c2f89-c8a1-490a-89e3-b1735b4ae271","Type":"ContainerDied","Data":"4f3b78fdef642d2f1bbeaa6914b68581e486e25e94d3e5a3bd2d545b122bec84"}
Nov 03 20:04:46.080039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.079710    5424 scope.go:122] "RemoveContainer" containerID="6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"
Nov 03 20:04:46.087711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.087690    5424 scope.go:122] "RemoveContainer" containerID="6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"
Nov 03 20:04:46.088016 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:46.087995    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c\": container with ID starting with 6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c not found: ID does not exist" containerID="6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"
Nov 03 20:04:46.088089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.088032    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c"} err="failed to get container status \"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c\": rpc error: code = NotFound desc = could not find container \"6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c\": container with ID starting with 6f6150b91e104ec5fae85e0128548b5ff3f5f69661d607a135b13bc138c16d3c not found: ID does not exist"
Nov 03 20:04:46.131277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.131186    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:04:46.144150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:46.144115    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7df975c8fb-xmwfw"]
Nov 03 20:04:47.917007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:47.916973    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811c2f89-c8a1-490a-89e3-b1735b4ae271" path="/var/lib/kubelet/pods/811c2f89-c8a1-490a-89e3-b1735b4ae271/volumes"
Nov 03 20:04:51.411923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.411887    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:51.412325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412272    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="prometheus" containerID="cri-o://c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e" gracePeriod=600
Nov 03 20:04:51.412404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412323    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-web" containerID="cri-o://06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff" gracePeriod=600
Nov 03 20:04:51.412404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412320    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7" gracePeriod=600
Nov 03 20:04:51.412958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412492    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy" containerID="cri-o://7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76" gracePeriod=600
Nov 03 20:04:51.412958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412802    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="thanos-sidecar" containerID="cri-o://68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836" gracePeriod=600
Nov 03 20:04:51.412958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.412919    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="config-reloader" containerID="cri-o://b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795" gracePeriod=600
Nov 03 20:04:51.695841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.695788    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:51.789588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789543    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.789786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789610    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.789786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789647    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.789786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789677    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.789786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789719    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.789786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789754    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789788    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789826    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789874    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789921    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789950    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.789985    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790030    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790069    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790065    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790103    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790131    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790156    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790168    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790252    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790284    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.790506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790316    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") pod \"0c9a2385-1990-48c4-a25c-6a2be45bb591\" (UID: \"0c9a2385-1990-48c4-a25c-6a2be45bb591\") "
Nov 03 20:04:51.791035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790611    5424 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.791035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.790631    5424 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.791137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.791044    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.791252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.791229    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.791866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.791603    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.793275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.793245    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.793382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.793280    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.797156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.797124    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.797288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.797230    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.797582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.797557    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:51.797929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.797905    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798114    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:51.798305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798263    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798343    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798385    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "kube-api-access-hsbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:51.798420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798405    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798443    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.798911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.798887    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:51.810451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.810406    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config" pod "0c9a2385-1990-48c4-a25c-6a2be45bb591" (UID: "0c9a2385-1990-48c4-a25c-6a2be45bb591"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.891287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891249    5424 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891301    5424 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-web-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891316    5424 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-tls-assets\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891327    5424 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891338    5424 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891350    5424 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-db\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891361    5424 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891374    5424 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-grpc-tls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891386    5424 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c9a2385-1990-48c4-a25c-6a2be45bb591-config-out\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891397    5424 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-kube-rbac-proxy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891409    5424 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891421    5424 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-metrics-client-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891434    5424 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891446    5424 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c9a2385-1990-48c4-a25c-6a2be45bb591-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891458    5424 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-metrics-client-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891470    5424 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891483    5424 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c9a2385-1990-48c4-a25c-6a2be45bb591-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:51.891971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:51.891496    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbzh\" (UniqueName: \"kubernetes.io/projected/0c9a2385-1990-48c4-a25c-6a2be45bb591-kube-api-access-hsbzh\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103755    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7" exitCode=0
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103781    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76" exitCode=0
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103790    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff" exitCode=0
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103800    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836" exitCode=0
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103807    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795" exitCode=0
Nov 03 20:04:52.103802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103814    5424 generic.go:356] "Generic (PLEG): container finished" podID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e" exitCode=0
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103843    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103875    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103935    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103978    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.103999    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.104014    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.104057    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.104098    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"}
Nov 03 20:04:52.106290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.104138    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c9a2385-1990-48c4-a25c-6a2be45bb591","Type":"ContainerDied","Data":"c91309eeefce47471de43bea253dc3cc15b23dc9a368ed72ae05c0cb26e50abf"}
Nov 03 20:04:52.113358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.113338    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.120604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.120576    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.128150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.128125    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.134960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.134943    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.141941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.141921    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.148827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.148799    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.155380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.155363    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.155648 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.155630    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.155703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.155663    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.155703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.155686    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.155945 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.155926    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.156018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.155951    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.156018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.155968    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.156207 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.156191    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.156253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156211    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.156253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156224    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.156466 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.156450    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.156510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156471    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.156510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156484    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.156754 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.156737    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.156815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156756    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.156815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156767    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.156984 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.156965    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.157050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.156990    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.157050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.157006    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.157478 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:04:52.157443    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.157612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.157469    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.157612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.157504    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.157927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.157904    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.158020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.157928    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.158329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158312    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.158399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158330    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.158610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158589    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.158610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158609    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.158839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158821    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.158887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.158842    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.159116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159086    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.159116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159107    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.159349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159331    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.159395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159350    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.159569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159553    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.159612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159569    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.159785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159768    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.159854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.159787    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.160068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160052    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.160108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160070    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.160241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160224    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.160300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160242    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.160468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160445    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.160515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160471    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.160726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160707    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.160770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160728    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.160972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160951    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.161020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.160974    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.161055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161019    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:52.161220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161204    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.161259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161220    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.161425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161409    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.161473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161425    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.161674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161654    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.161745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161675    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.161929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161911    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.161978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.161929    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.162160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162143    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.162200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162161    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.162362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162348    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.162362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162362    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.162596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162579    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.162596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162595    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.162801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162783    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.162801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.162797    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.163021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163004    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.163063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163022    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.163256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163240    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.163256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163255    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.163463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163447    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.163463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163463    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.163696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163674    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.163779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163697    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.163927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163912    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.163927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.163926    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.164128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164115    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.164128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164127    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.164326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164312    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.164326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164325    5424 scope.go:122] "RemoveContainer" containerID="a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"
Nov 03 20:04:52.164550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164516    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7"} err="failed to get container status \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": rpc error: code = NotFound desc = could not find container \"a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7\": container with ID starting with a16b3908d2e0859461f0d82c690bba785b4daab0eb89b76cab54506a745c04c7 not found: ID does not exist"
Nov 03 20:04:52.164550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164543    5424 scope.go:122] "RemoveContainer" containerID="7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"
Nov 03 20:04:52.164738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164723    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76"} err="failed to get container status \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": rpc error: code = NotFound desc = could not find container \"7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76\": container with ID starting with 7dca90af5a20a0310f5b6f84af34401ed5578bfacb3afb62cc93a684f5219f76 not found: ID does not exist"
Nov 03 20:04:52.164778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164738    5424 scope.go:122] "RemoveContainer" containerID="06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"
Nov 03 20:04:52.164936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164919    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff"} err="failed to get container status \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": rpc error: code = NotFound desc = could not find container \"06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff\": container with ID starting with 06bdd3f2aa3c42df4a9fee4eff8b7c6aa16de274478f4639a17881a223878cff not found: ID does not exist"
Nov 03 20:04:52.165001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.164938    5424 scope.go:122] "RemoveContainer" containerID="68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"
Nov 03 20:04:52.165129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165111    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836"} err="failed to get container status \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": rpc error: code = NotFound desc = could not find container \"68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836\": container with ID starting with 68a070ca554423179757d0f511ad577a1d6fffbaa137c12cae9e3a8242ecc836 not found: ID does not exist"
Nov 03 20:04:52.165173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165131    5424 scope.go:122] "RemoveContainer" containerID="b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"
Nov 03 20:04:52.165306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165287    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795"} err="failed to get container status \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": rpc error: code = NotFound desc = could not find container \"b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795\": container with ID starting with b58cfe49c107b5ac69966e7c109fe2a1e0eb7bbae05fb45f51aa77cb5f2bc795 not found: ID does not exist"
Nov 03 20:04:52.165353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165307    5424 scope.go:122] "RemoveContainer" containerID="c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"
Nov 03 20:04:52.165506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165488    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e"} err="failed to get container status \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": rpc error: code = NotFound desc = could not find container \"c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e\": container with ID starting with c537678db64362d57ea102d525882e7fd84fe2cee8106d3b37a0c73b812ee25e not found: ID does not exist"
Nov 03 20:04:52.165632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165507    5424 scope.go:122] "RemoveContainer" containerID="1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"
Nov 03 20:04:52.165763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.165744    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736"} err="failed to get container status \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": rpc error: code = NotFound desc = could not find container \"1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736\": container with ID starting with 1abdf0911b282fa24cd4addeaebcdc74899bcf76191e81663e8fbfae79bab736 not found: ID does not exist"
Nov 03 20:04:52.173821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.173769    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:52.191668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191644    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:52.191872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191860    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191874    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191884    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="thanos-sidecar"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191888    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="thanos-sidecar"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191895    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="prometheus"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191899    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="prometheus"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191907    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-web"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191911    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-web"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191918    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="config-reloader"
Nov 03 20:04:52.191921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191922    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="config-reloader"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191932    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-thanos"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191936    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-thanos"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191945    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="init-config-reloader"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191949    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="init-config-reloader"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191956    5424 cpu_manager.go:414] "containerMap: removing container" podUID="811c2f89-c8a1-490a-89e3-b1735b4ae271" containerName="console"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191960    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811c2f89-c8a1-490a-89e3-b1735b4ae271" containerName="console"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.191997    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="811c2f89-c8a1-490a-89e3-b1735b4ae271" containerName="console"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192005    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192011    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="thanos-sidecar"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192016    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="prometheus"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192021    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-web"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192026    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="config-reloader"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192031    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="kube-rbac-proxy-thanos"
Nov 03 20:04:52.192180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.192036    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" containerName="init-config-reloader"
Nov 03 20:04:52.224791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.224748    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:52.224949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.224896    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.241024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.240994    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:04:52.241024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.240995    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 20:04:52.241024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241001    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 20:04:52.241281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241005    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 20:04:52.241281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241194    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 20:04:52.241281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241203    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 20:04:52.241281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241234    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 20:04:52.241281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241265    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 20:04:52.241464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241312    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 20:04:52.241464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241316    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xglnv\""
Nov 03 20:04:52.241464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241333    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 20:04:52.241464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241326    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 20:04:52.241464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241379    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 20:04:52.241659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241502    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3apnarc55k39g\""
Nov 03 20:04:52.241659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.241558    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 20:04:52.256152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.256114    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 20:04:52.259354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.259332    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 20:04:52.295615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295585    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295620    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295648    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295764    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295790    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295811    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295855    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295884    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295952    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.295986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295972    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.295985    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296025    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296057    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296126 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296091    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296126 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296114    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296133    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hj24\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-kube-api-access-8hj24\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296187    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296210    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296227    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.296315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.296256    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.396849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396807    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.396849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396850    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396874    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396900    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396925    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396947    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.396986    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397006    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397026    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397047    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397080    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397101    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397118    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397140    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hj24\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-kube-api-access-8hj24\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397167    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397190    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397213    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397216    5424 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397245    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397270    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397298    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.397601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.397318    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.398336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.398030    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.398389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.398331    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.399784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.398749    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.399784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.399364    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.399784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.399744    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.400183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.400044    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.400797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.400754    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8583e6da-46d9-4a06-ab84-7714426042cf-config-out\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.401304    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.401793    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402025    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402116    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402192    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402227    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402450    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8583e6da-46d9-4a06-ab84-7714426042cf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.402655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.402454    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-web-config\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.403193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.403173    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.403270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.403247    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.403270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.403253    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8583e6da-46d9-4a06-ab84-7714426042cf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.418153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.418131    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hj24\" (UniqueName: \"kubernetes.io/projected/8583e6da-46d9-4a06-ab84-7714426042cf-kube-api-access-8hj24\") pod \"prometheus-k8s-0\" (UID: \"8583e6da-46d9-4a06-ab84-7714426042cf\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.625979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.625936    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:04:52.749905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.749859    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5vbqn"
Nov 03 20:04:52.974819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:52.974794    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 20:04:52.976019 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:52.975990    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8583e6da_46d9_4a06_ab84_7714426042cf.slice/crio-da8d14f71e2fc645a83e9ec40db7a0dbcf29016e45696bcf04fed19225db244a WatchSource:0}: Error finding container da8d14f71e2fc645a83e9ec40db7a0dbcf29016e45696bcf04fed19225db244a: Status 404 returned error can't find the container with id da8d14f71e2fc645a83e9ec40db7a0dbcf29016e45696bcf04fed19225db244a
Nov 03 20:04:53.109149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:53.109120    5424 generic.go:356] "Generic (PLEG): container finished" podID="8583e6da-46d9-4a06-ab84-7714426042cf" containerID="11ef6eb00e35aaad137055916de4152f31d0edee36f77a285ec9288e323856c5" exitCode=0
Nov 03 20:04:53.109149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:53.109154    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerDied","Data":"11ef6eb00e35aaad137055916de4152f31d0edee36f77a285ec9288e323856c5"}
Nov 03 20:04:53.109400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:53.109202    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"da8d14f71e2fc645a83e9ec40db7a0dbcf29016e45696bcf04fed19225db244a"}
Nov 03 20:04:53.919721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:53.916971    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9a2385-1990-48c4-a25c-6a2be45bb591" path="/var/lib/kubelet/pods/0c9a2385-1990-48c4-a25c-6a2be45bb591/volumes"
Nov 03 20:04:54.115410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115372    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"8f3047f1dd194e8a37038615785049bfbf0e68aa6eb4060c38fa0504ef2aae74"}
Nov 03 20:04:54.115600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115452    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"d12c70efcb1d5f735708fd5496e646fb516cdd901b6544401f5fcc9d404e249c"}
Nov 03 20:04:54.115600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115496    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"6b4bcf4855ea62cf1aaf5c3f5750ed006e5d8c12023529103527d3c7ed36ef55"}
Nov 03 20:04:54.115600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115550    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"28e057756951ea976ea559f2578f594fd74a974e6842face0d7609b319c6f1b6"}
Nov 03 20:04:54.115600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115584    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"7df68726f6ae68170df36070ac11360a385863cce8dae2d8ae3729856d35744a"}
Nov 03 20:04:54.115726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.115623    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8583e6da-46d9-4a06-ab84-7714426042cf","Type":"ContainerStarted","Data":"70f9a91692ca5c37b0ab956e66fd9d42f12e6dceac18d1518b816ecd6c6d52dc"}
Nov 03 20:04:54.165463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.165406    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.150792674 podStartE2EDuration="2.165392435s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.599761ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:54.164263191 +0000 UTC m=+36.373279186" watchObservedRunningTime="2026-11-03 20:04:54.165392435 +0000 UTC m=+36.374408423"
Nov 03 20:04:54.347418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.347387    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:04:54.350127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.350102    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.370996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.370957    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:04:54.415379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415332    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415378    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415410    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415520    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczsz\" (UniqueName: \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415578    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415610    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415625    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.415675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.415644    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516782    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516819    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516847    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczsz\" (UniqueName: \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516863    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516885    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516907    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.516934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516931    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.516978    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.517339    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.517679    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.517729    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.517814    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.517844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.517817    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.519520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.519499    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.519694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.519681    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.537753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.537730    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczsz\" (UniqueName: \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\") pod \"console-66f5844b87-n7sg6\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") " pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.661021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.660932    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:04:54.955157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:54.955133    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:04:54.956714 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:04:54.956693    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736c3097_a877_4542_938c_c6ff88356317.slice/crio-85312bf1ac34dc456d0d4f70d6e418393482c494ed5fd6d950b76bd9f3b0dfe6 WatchSource:0}: Error finding container 85312bf1ac34dc456d0d4f70d6e418393482c494ed5fd6d950b76bd9f3b0dfe6: Status 404 returned error can't find the container with id 85312bf1ac34dc456d0d4f70d6e418393482c494ed5fd6d950b76bd9f3b0dfe6
Nov 03 20:04:55.120297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:55.120264    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5844b87-n7sg6" event={"ID":"736c3097-a877-4542-938c-c6ff88356317","Type":"ContainerStarted","Data":"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"}
Nov 03 20:04:55.120482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:55.120316    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5844b87-n7sg6" event={"ID":"736c3097-a877-4542-938c-c6ff88356317","Type":"ContainerStarted","Data":"85312bf1ac34dc456d0d4f70d6e418393482c494ed5fd6d950b76bd9f3b0dfe6"}
Nov 03 20:04:55.160756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:55.160703    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66f5844b87-n7sg6" podStartSLOduration=1.160688326 podStartE2EDuration="1.160688326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:55.160031559 +0000 UTC m=+37.369047590" watchObservedRunningTime="2026-11-03 20:04:55.160688326 +0000 UTC m=+37.369704318"
Nov 03 20:04:57.626311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:04:57.626254    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:05:04.661048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:04.661003    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:05:04.661048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:04.661070    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:05:04.665780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:04.665757    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:05:05.157813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:05.157786    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:05:05.222278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:05.222244    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:05:12.285548 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:05:12.285494    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:05:30.242272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.242212    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66fcdcfbdb-6hb6r" podUID="e062a8d2-774a-4cf4-a741-bf91149d8d90" containerName="console" containerID="cri-o://21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671" gracePeriod=15
Nov 03 20:05:30.487322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.487298    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:05:30.612308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612219    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612274    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612315    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612348    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612384    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612420    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612579    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612634    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") pod \"e062a8d2-774a-4cf4-a741-bf91149d8d90\" (UID: \"e062a8d2-774a-4cf4-a741-bf91149d8d90\") "
Nov 03 20:05:30.612763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612669    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:30.612763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612674    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:30.612991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612903    5424 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.612991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612923    5424 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e062a8d2-774a-4cf4-a741-bf91149d8d90-tmp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.612991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612929    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:30.612991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.612959    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:30.613130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.613046    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:30.614843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.614818    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:30.614936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.614844    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:30.614936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.614853    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7" pod "e062a8d2-774a-4cf4-a741-bf91149d8d90" (UID: "e062a8d2-774a-4cf4-a741-bf91149d8d90"). InnerVolumeSpecName "kube-api-access-d5vp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:30.713917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713876    5424 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.713917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713910    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vp7\" (UniqueName: \"kubernetes.io/projected/e062a8d2-774a-4cf4-a741-bf91149d8d90-kube-api-access-d5vp7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.713917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713920    5424 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-oauth-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.713917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713929    5424 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-oauth-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.714179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713937    5424 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e062a8d2-774a-4cf4-a741-bf91149d8d90-trusted-ca-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:30.714179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:30.713945    5424 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e062a8d2-774a-4cf4-a741-bf91149d8d90-console-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:05:31.242013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.241977    5424 generic.go:356] "Generic (PLEG): container finished" podID="e062a8d2-774a-4cf4-a741-bf91149d8d90" containerID="21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671" exitCode=0
Nov 03 20:05:31.242013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.242014    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fcdcfbdb-6hb6r" event={"ID":"e062a8d2-774a-4cf4-a741-bf91149d8d90","Type":"ContainerDied","Data":"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"}
Nov 03 20:05:31.242247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.242039    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fcdcfbdb-6hb6r"
Nov 03 20:05:31.242247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.242049    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fcdcfbdb-6hb6r" event={"ID":"e062a8d2-774a-4cf4-a741-bf91149d8d90","Type":"ContainerDied","Data":"fc8eded88344ae5761b16cf03e01b444f183da5a2e262019573672a3aad623a6"}
Nov 03 20:05:31.242247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.242069    5424 scope.go:122] "RemoveContainer" containerID="21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"
Nov 03 20:05:31.250631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.250611    5424 scope.go:122] "RemoveContainer" containerID="21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"
Nov 03 20:05:31.250920 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:05:31.250899    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671\": container with ID starting with 21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671 not found: ID does not exist" containerID="21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"
Nov 03 20:05:31.250971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.250931    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671"} err="failed to get container status \"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671\": rpc error: code = NotFound desc = could not find container \"21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671\": container with ID starting with 21573b2c69b9cb7e72dd5c6426a4cc8f80f09e93f73baa204b8c5494272f8671 not found: ID does not exist"
Nov 03 20:05:31.293418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.293388    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:05:31.306190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.306161    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66fcdcfbdb-6hb6r"]
Nov 03 20:05:31.383482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.383437    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:05:31.399329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.399303    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:05:31.916505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:31.916475    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e062a8d2-774a-4cf4-a741-bf91149d8d90" path="/var/lib/kubelet/pods/e062a8d2-774a-4cf4-a741-bf91149d8d90/volumes"
Nov 03 20:05:32.262204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:32.262175    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 20:05:42.284967 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:05:42.284930    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:05:50.755502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.755469    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5974f9cd-t9czm"]
Nov 03 20:05:50.755897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.755770    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e062a8d2-774a-4cf4-a741-bf91149d8d90" containerName="console"
Nov 03 20:05:50.755897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.755782    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e062a8d2-774a-4cf4-a741-bf91149d8d90" containerName="console"
Nov 03 20:05:50.755897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.755821    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e062a8d2-774a-4cf4-a741-bf91149d8d90" containerName="console"
Nov 03 20:05:50.758710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.758690    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761224    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761258    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-tmp\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761336    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpzr\" (UniqueName: \"kubernetes.io/projected/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-kube-api-access-7wpzr\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761370    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-oauth-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761392    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-oauth-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761412    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761482    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-trusted-ca-bundle\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.761583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.761558    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-service-ca\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.779070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.779039    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974f9cd-t9czm"]
Nov 03 20:05:50.862398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862361    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862402    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-tmp\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862432    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpzr\" (UniqueName: \"kubernetes.io/projected/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-kube-api-access-7wpzr\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862450    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-oauth-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862471    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-oauth-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862500    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862549    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-trusted-ca-bundle\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.862763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.862579    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-service-ca\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.863281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.863202    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-tmp\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.868331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.863624    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-service-ca\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.868331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.863902    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.868331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.864131    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-oauth-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.868331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.864458    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-trusted-ca-bundle\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.868331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.866471    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-serving-cert\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.870218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.870188    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-console-oauth-config\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:50.882621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:50.882594    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpzr\" (UniqueName: \"kubernetes.io/projected/0d6e6d6c-0149-4036-b9c3-093b2c6f417a-kube-api-access-7wpzr\") pod \"console-5974f9cd-t9czm\" (UID: \"0d6e6d6c-0149-4036-b9c3-093b2c6f417a\") " pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:51.069400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:51.069316    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:05:51.361366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:51.361342    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974f9cd-t9czm"]
Nov 03 20:05:51.363107 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:05:51.363076    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6e6d6c_0149_4036_b9c3_093b2c6f417a.slice/crio-ca2515c14f729aba22b5b85bcabdc5650f2e7edf30a30ba6b77210cd13957a9c WatchSource:0}: Error finding container ca2515c14f729aba22b5b85bcabdc5650f2e7edf30a30ba6b77210cd13957a9c: Status 404 returned error can't find the container with id ca2515c14f729aba22b5b85bcabdc5650f2e7edf30a30ba6b77210cd13957a9c
Nov 03 20:05:52.313955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:52.313914    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974f9cd-t9czm" event={"ID":"0d6e6d6c-0149-4036-b9c3-093b2c6f417a","Type":"ContainerStarted","Data":"8f04ffbbc9082a2f46fd64dc1a9a6188306e3c00d1c25566e16aa24f0cc5f4d1"}
Nov 03 20:05:52.313955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:52.313964    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974f9cd-t9czm" event={"ID":"0d6e6d6c-0149-4036-b9c3-093b2c6f417a","Type":"ContainerStarted","Data":"ca2515c14f729aba22b5b85bcabdc5650f2e7edf30a30ba6b77210cd13957a9c"}
Nov 03 20:05:52.366923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:05:52.366863    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5974f9cd-t9czm" podStartSLOduration=2.366837233 podStartE2EDuration="2.366837233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:52.353918701 +0000 UTC m=+94.562934698" watchObservedRunningTime="2026-11-03 20:05:52.366837233 +0000 UTC m=+94.575853224"
Nov 03 20:06:01.069516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:01.069431    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:06:01.069516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:01.069512    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:06:01.074468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:01.074443    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:06:01.345331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:01.345258    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5974f9cd-t9czm"
Nov 03 20:06:01.414430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:01.414389    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:06:12.285166 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:06:12.285125    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:06:26.434434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.434374    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66f5844b87-n7sg6" podUID="736c3097-a877-4542-938c-c6ff88356317" containerName="console" containerID="cri-o://41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08" gracePeriod=15
Nov 03 20:06:26.667421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.667396    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:06:26.747524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747493    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\" (UniqueName: \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747552    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747619    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747638    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747677    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\" (UniqueName: \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747815    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:26.747894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747827    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.747894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747875    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.748036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.747906    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\") pod \"736c3097-a877-4542-938c-c6ff88356317\" (UID: \"736c3097-a877-4542-938c-c6ff88356317\") "
Nov 03 20:06:26.748085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748067    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.748138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748081    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.748223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748198    5424 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736c3097-a877-4542-938c-c6ff88356317-tmp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.748384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748192    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.748384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748336    5424 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.748384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748357    5424 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-oauth-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.748545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.748385    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.750178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.750159    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:26.750485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.750465    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:26.750546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.750482    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz" pod "736c3097-a877-4542-938c-c6ff88356317" (UID: "736c3097-a877-4542-938c-c6ff88356317"). InnerVolumeSpecName "kube-api-access-bczsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:26.849222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.849166    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bczsz\" (UniqueName: \"kubernetes.io/projected/736c3097-a877-4542-938c-c6ff88356317-kube-api-access-bczsz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.849222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.849215    5424 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-trusted-ca-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.849222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.849226    5424 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-oauth-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.849222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.849236    5424 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/736c3097-a877-4542-938c-c6ff88356317-console-config\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:26.849495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:26.849247    5424 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/736c3097-a877-4542-938c-c6ff88356317-console-serving-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:06:27.430134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.430095    5424 generic.go:356] "Generic (PLEG): container finished" podID="736c3097-a877-4542-938c-c6ff88356317" containerID="41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08" exitCode=0
Nov 03 20:06:27.430134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.430134    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5844b87-n7sg6" event={"ID":"736c3097-a877-4542-938c-c6ff88356317","Type":"ContainerDied","Data":"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"}
Nov 03 20:06:27.430389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.430150    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f5844b87-n7sg6"
Nov 03 20:06:27.430389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.430182    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f5844b87-n7sg6" event={"ID":"736c3097-a877-4542-938c-c6ff88356317","Type":"ContainerDied","Data":"85312bf1ac34dc456d0d4f70d6e418393482c494ed5fd6d950b76bd9f3b0dfe6"}
Nov 03 20:06:27.430389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.430203    5424 scope.go:122] "RemoveContainer" containerID="41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"
Nov 03 20:06:27.438735 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.438600    5424 scope.go:122] "RemoveContainer" containerID="41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"
Nov 03 20:06:27.438962 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:06:27.438894    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08\": container with ID starting with 41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08 not found: ID does not exist" containerID="41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"
Nov 03 20:06:27.438962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.438917    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08"} err="failed to get container status \"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08\": rpc error: code = NotFound desc = could not find container \"41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08\": container with ID starting with 41c84b60577c9fa7f970cca827d68866618fc9cc6a5df43b267e7a868e158a08 not found: ID does not exist"
Nov 03 20:06:27.481373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.481345    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:06:27.494295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.494266    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f5844b87-n7sg6"]
Nov 03 20:06:27.916295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:06:27.916261    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736c3097-a877-4542-938c-c6ff88356317" path="/var/lib/kubelet/pods/736c3097-a877-4542-938c-c6ff88356317/volumes"
Nov 03 20:06:42.287360 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:06:42.287323    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:07:12.286269 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:07:12.286224    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:07:42.285205 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:07:42.285165    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:08:12.284698 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:08:12.284662    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:08:42.284325 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:08:42.284286    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:09:12.284088 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:09:12.284008    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:09:42.286570 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:09:42.286502    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:09:54.543191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.543149    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"]
Nov 03 20:09:54.543824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.543432    5424 cpu_manager.go:414] "containerMap: removing container" podUID="736c3097-a877-4542-938c-c6ff88356317" containerName="console"
Nov 03 20:09:54.543824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.543443    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736c3097-a877-4542-938c-c6ff88356317" containerName="console"
Nov 03 20:09:54.543824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.543505    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="736c3097-a877-4542-938c-c6ff88356317" containerName="console"
Nov 03 20:09:54.546252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.546232    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.574082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.574037    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:09:54.574082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.574050    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:09:54.574349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.574185    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-76jdb\""
Nov 03 20:09:54.574757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.574733    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"]
Nov 03 20:09:54.603472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.603406    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.603699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.603560    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.603699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.603641    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4r76\" (UniqueName: \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.705108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.705062    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.705108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.705116    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.705375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.705158    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4r76\" (UniqueName: \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.705498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.705477    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.705610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.705504    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.726337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.726293    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4r76\" (UniqueName: \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:54.862620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:54.862495    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:09:55.159543 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.159505    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"]
Nov 03 20:09:55.161832 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:55.161804    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f9ed64_e961_4d13_99c7_92b76d0e0924.slice/crio-4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8 WatchSource:0}: Error finding container 4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8: Status 404 returned error can't find the container with id 4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8
Nov 03 20:09:55.163831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.163813    5424 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:09:55.811698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.811663    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"]
Nov 03 20:09:55.814945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.814920    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:55.845017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.844975    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-th68q\""
Nov 03 20:09:55.845667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.845639    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"]
Nov 03 20:09:55.915709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.915662    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrst\" (UniqueName: \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:55.915709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.915711    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:55.915972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:55.915831    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.016909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.016868    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrst\" (UniqueName: \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.017098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.016921    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.017098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.016957    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.017410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.017384    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.017557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.017445    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.042217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.042175    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"]
Nov 03 20:09:56.046112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.046087    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.050150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.050124    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrst\" (UniqueName: \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.061420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.061222    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-xbrx9\""
Nov 03 20:09:56.065200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.065144    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"]
Nov 03 20:09:56.112686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.112647    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerStarted","Data":"4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8"}
Nov 03 20:09:56.118176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.118140    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhpr\" (UniqueName: \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.118403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.118320    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.118403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.118353    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.133781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.133746    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:09:56.159647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.159603    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"]
Nov 03 20:09:56.163402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.163377    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.184161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.184088    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-z5zjn\""
Nov 03 20:09:56.185360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.185334    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"]
Nov 03 20:09:56.219139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219110    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.219311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219146    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.219311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219171    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.219311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219193    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhpr\" (UniqueName: \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.219311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219215    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvtm\" (UniqueName: \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.219311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219247    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.219907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219612    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.219907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.219642    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.244590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.243278    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"]
Nov 03 20:09:56.247212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.246811    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.249578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.249198    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhpr\" (UniqueName: \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.269239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.269211    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"]
Nov 03 20:09:56.281983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.281945    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-bbv7k\""
Nov 03 20:09:56.320470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320373    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.320470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320456    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.320751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320496    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvjk\" (UniqueName: \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.320751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320566    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.320751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320599    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.320751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320661    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvtm\" (UniqueName: \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.320949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320822    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.320949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.320888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.347005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.346936    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvtm\" (UniqueName: \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.371117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.370861    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:09:56.421569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.421474    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.421569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.421547    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvjk\" (UniqueName: \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.421797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.421588    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.422061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.422041    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.422307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.422291    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.443053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.443022    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvjk\" (UniqueName: \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.456932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.456908    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"]
Nov 03 20:09:56.459025 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:56.458994    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6a52c3_091a_4275_b219_138d180bc720.slice/crio-b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657 WatchSource:0}: Error finding container b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657: Status 404 returned error can't find the container with id b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657
Nov 03 20:09:56.486242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.486210    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:09:56.574919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.574848    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:09:56.674999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.674966    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"]
Nov 03 20:09:56.676742 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:56.676712    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4daec27_811e_4e06_830d_5a63e15f7f66.slice/crio-88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b WatchSource:0}: Error finding container 88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b: Status 404 returned error can't find the container with id 88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b
Nov 03 20:09:56.788480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.788450    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"]
Nov 03 20:09:56.789676 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:56.789632    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e85217_5ecd_4a0f_9842_daa0ed1b5c19.slice/crio-9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da WatchSource:0}: Error finding container 9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da: Status 404 returned error can't find the container with id 9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da
Nov 03 20:09:56.878878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:56.878132    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"]
Nov 03 20:09:56.880886 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:56.880852    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75832162_9744_43f9_84b4_2ccbf4a4bd78.slice/crio-8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c WatchSource:0}: Error finding container 8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c: Status 404 returned error can't find the container with id 8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c
Nov 03 20:09:57.118871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:57.118840    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerStarted","Data":"88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b"}
Nov 03 20:09:57.120844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:57.120812    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr" event={"ID":"75832162-9744-43f9-84b4-2ccbf4a4bd78","Type":"ContainerStarted","Data":"8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c"}
Nov 03 20:09:57.123605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:57.123577    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t" event={"ID":"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19","Type":"ContainerStarted","Data":"9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da"}
Nov 03 20:09:57.125432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:57.125399    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48" event={"ID":"9d6a52c3-091a-4275-b219-138d180bc720","Type":"ContainerStarted","Data":"b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657"}
Nov 03 20:09:57.999047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:57.997271    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"]
Nov 03 20:09:58.001890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.001215    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.030224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.030189    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-t2v8n\""
Nov 03 20:09:58.030951 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.030926    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"]
Nov 03 20:09:58.140205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.140168    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.140387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.140270    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.140387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.140327    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2wj\" (UniqueName: \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.242406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.241762    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2wj\" (UniqueName: \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.242406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.241835    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.242406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.241906    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.242406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.242329    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.242780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.242495    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.278233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.278064    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2wj\" (UniqueName: \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.323913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.323825    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:09:58.647105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:58.646937    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"]
Nov 03 20:09:58.654803 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:09:58.654762    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748b270c_46d1_450d_a8e6_44e8c4ecbcd2.slice/crio-e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5 WatchSource:0}: Error finding container e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5: Status 404 returned error can't find the container with id e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5
Nov 03 20:09:59.136856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:09:59.136820    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9" event={"ID":"748b270c-46d1-450d-a8e6-44e8c4ecbcd2","Type":"ContainerStarted","Data":"e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5"}
Nov 03 20:10:06.184072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:06.184026    5424 generic.go:356] "Generic (PLEG): container finished" podID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerID="7818be16608363d9e0461d81c3d66677ce05857481e22be3d738637a4fb1fa88" exitCode=0
Nov 03 20:10:06.184571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:06.184148    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerDied","Data":"7818be16608363d9e0461d81c3d66677ce05857481e22be3d738637a4fb1fa88"}
Nov 03 20:10:07.194659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:07.194616    5424 generic.go:356] "Generic (PLEG): container finished" podID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerID="b6ec93ceec44f24ec7f17cc07a16c37953040a7532179065b59c3883b79ccb71" exitCode=0
Nov 03 20:10:07.195093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:07.194708    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9" event={"ID":"748b270c-46d1-450d-a8e6-44e8c4ecbcd2","Type":"ContainerDied","Data":"b6ec93ceec44f24ec7f17cc07a16c37953040a7532179065b59c3883b79ccb71"}
Nov 03 20:10:08.200724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:08.200685    5424 generic.go:356] "Generic (PLEG): container finished" podID="9d6a52c3-091a-4275-b219-138d180bc720" containerID="febb9260c16be1450d07c3a94d5fcd81e43c61c3f7fa8dd59ccb25e8defd8c8c" exitCode=0
Nov 03 20:10:08.201171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:08.200766    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48" event={"ID":"9d6a52c3-091a-4275-b219-138d180bc720","Type":"ContainerDied","Data":"febb9260c16be1450d07c3a94d5fcd81e43c61c3f7fa8dd59ccb25e8defd8c8c"}
Nov 03 20:10:08.203402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:08.203379    5424 generic.go:356] "Generic (PLEG): container finished" podID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerID="1de75e5049849015671f928dddabdaa31209fd212416b9d4629f15f1bb84a117" exitCode=0
Nov 03 20:10:08.203484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:08.203423    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerDied","Data":"1de75e5049849015671f928dddabdaa31209fd212416b9d4629f15f1bb84a117"}
Nov 03 20:10:09.209396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.209306    5424 generic.go:356] "Generic (PLEG): container finished" podID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerID="a2bb913080acfc1049b1e4fe431542339a5b8ad0b94ec816d7b633af6fe81127" exitCode=0
Nov 03 20:10:09.209840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.209400    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t" event={"ID":"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19","Type":"ContainerDied","Data":"a2bb913080acfc1049b1e4fe431542339a5b8ad0b94ec816d7b633af6fe81127"}
Nov 03 20:10:09.213393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.213364    5424 generic.go:356] "Generic (PLEG): container finished" podID="9d6a52c3-091a-4275-b219-138d180bc720" containerID="c4a648eb4da452cf58868567a1dcf7743d007147ff1c76fcb5fc6c461a87584d" exitCode=0
Nov 03 20:10:09.213685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.213657    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48" event={"ID":"9d6a52c3-091a-4275-b219-138d180bc720","Type":"ContainerDied","Data":"c4a648eb4da452cf58868567a1dcf7743d007147ff1c76fcb5fc6c461a87584d"}
Nov 03 20:10:09.216940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.216914    5424 generic.go:356] "Generic (PLEG): container finished" podID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerID="523dc08d9c00a85a4888e71e9b86374e4a86d02f73e9baebe2ff5f9451a0dbc7" exitCode=0
Nov 03 20:10:09.217036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.216998    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerDied","Data":"523dc08d9c00a85a4888e71e9b86374e4a86d02f73e9baebe2ff5f9451a0dbc7"}
Nov 03 20:10:09.221175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.220474    5424 generic.go:356] "Generic (PLEG): container finished" podID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerID="b7317a5a377433399c16826e285a237a1fa33afd8b7e59e70238ff3e9affb2ca" exitCode=0
Nov 03 20:10:09.221175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:09.220565    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr" event={"ID":"75832162-9744-43f9-84b4-2ccbf4a4bd78","Type":"ContainerDied","Data":"b7317a5a377433399c16826e285a237a1fa33afd8b7e59e70238ff3e9affb2ca"}
Nov 03 20:10:10.228257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:10.228063    5424 generic.go:356] "Generic (PLEG): container finished" podID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerID="74bada63a79b6edf6d057035be2b1c01f781a3cad58ac8fd718327e5cbfe1007" exitCode=0
Nov 03 20:10:10.228257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:10.228159    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t" event={"ID":"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19","Type":"ContainerDied","Data":"74bada63a79b6edf6d057035be2b1c01f781a3cad58ac8fd718327e5cbfe1007"}
Nov 03 20:10:10.233568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:10.233016    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerStarted","Data":"eef8787fd961255fc011b1b11866496917db7f4ed818752adb166bd1282d305d"}
Nov 03 20:10:10.238419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:10.238383    5424 generic.go:356] "Generic (PLEG): container finished" podID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerID="954907fc3c830514b6babee18f77f095c2278ff9a4d44cd851f6de0fed31dd1a" exitCode=0
Nov 03 20:10:10.238570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:10.238469    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr" event={"ID":"75832162-9744-43f9-84b4-2ccbf4a4bd78","Type":"ContainerDied","Data":"954907fc3c830514b6babee18f77f095c2278ff9a4d44cd851f6de0fed31dd1a"}
Nov 03 20:10:11.246250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:11.246214    5424 generic.go:356] "Generic (PLEG): container finished" podID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerID="eef8787fd961255fc011b1b11866496917db7f4ed818752adb166bd1282d305d" exitCode=0
Nov 03 20:10:11.246793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:11.246375    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerDied","Data":"eef8787fd961255fc011b1b11866496917db7f4ed818752adb166bd1282d305d"}
Nov 03 20:10:11.253240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:11.252423    5424 generic.go:356] "Generic (PLEG): container finished" podID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerID="ba89b9c567780247606629ac990920fedd15016f132df5d431074802f3e005ae" exitCode=0
Nov 03 20:10:11.253240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:11.252464    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9" event={"ID":"748b270c-46d1-450d-a8e6-44e8c4ecbcd2","Type":"ContainerDied","Data":"ba89b9c567780247606629ac990920fedd15016f132df5d431074802f3e005ae"}
Nov 03 20:10:12.314712 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:10:12.314668    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:10:20.326243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:20.326137    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerStarted","Data":"591acb59596e86d8f681e6066d58ada594dcf19797d1f26e4d8fe0782576ddb7"}
Nov 03 20:10:21.334661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:21.334625    5424 generic.go:356] "Generic (PLEG): container finished" podID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerID="591acb59596e86d8f681e6066d58ada594dcf19797d1f26e4d8fe0782576ddb7" exitCode=0
Nov 03 20:10:21.335049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:21.334703    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerDied","Data":"591acb59596e86d8f681e6066d58ada594dcf19797d1f26e4d8fe0782576ddb7"}
Nov 03 20:10:22.343139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.343102    5424 generic.go:356] "Generic (PLEG): container finished" podID="9d6a52c3-091a-4275-b219-138d180bc720" containerID="f767d67c9cd56fe332a6cb48e5a9cf120f1a76e6af3c340e7519c07f6378bc2b" exitCode=0
Nov 03 20:10:22.343852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.343212    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48" event={"ID":"9d6a52c3-091a-4275-b219-138d180bc720","Type":"ContainerDied","Data":"f767d67c9cd56fe332a6cb48e5a9cf120f1a76e6af3c340e7519c07f6378bc2b"}
Nov 03 20:10:22.520869 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.520562    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:10:22.593586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.593485    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\" (UniqueName: \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\") pod \"05f9ed64-e961-4d13-99c7-92b76d0e0924\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") "
Nov 03 20:10:22.593586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.593575    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\") pod \"05f9ed64-e961-4d13-99c7-92b76d0e0924\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") "
Nov 03 20:10:22.593818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.593600    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\") pod \"05f9ed64-e961-4d13-99c7-92b76d0e0924\" (UID: \"05f9ed64-e961-4d13-99c7-92b76d0e0924\") "
Nov 03 20:10:22.594780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.594733    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle" pod "05f9ed64-e961-4d13-99c7-92b76d0e0924" (UID: "05f9ed64-e961-4d13-99c7-92b76d0e0924"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:22.597833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.597804    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76" pod "05f9ed64-e961-4d13-99c7-92b76d0e0924" (UID: "05f9ed64-e961-4d13-99c7-92b76d0e0924"). InnerVolumeSpecName "kube-api-access-f4r76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:22.599616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.599585    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util" pod "05f9ed64-e961-4d13-99c7-92b76d0e0924" (UID: "05f9ed64-e961-4d13-99c7-92b76d0e0924"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:22.695165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.695126    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4r76\" (UniqueName: \"kubernetes.io/projected/05f9ed64-e961-4d13-99c7-92b76d0e0924-kube-api-access-f4r76\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:22.695165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.695163    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:22.695388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:22.695181    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05f9ed64-e961-4d13-99c7-92b76d0e0924-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:23.348221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.348188    5424 generic.go:356] "Generic (PLEG): container finished" podID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerID="3868278c79143ac511d8ce8b2f25acec40bebb1b326fedb01a57dcfc4cee6ad9" exitCode=0
Nov 03 20:10:23.348698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.348267    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t" event={"ID":"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19","Type":"ContainerDied","Data":"3868278c79143ac511d8ce8b2f25acec40bebb1b326fedb01a57dcfc4cee6ad9"}
Nov 03 20:10:23.350227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.350204    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6"
Nov 03 20:10:23.350346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.350224    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sk9v6" event={"ID":"05f9ed64-e961-4d13-99c7-92b76d0e0924","Type":"ContainerDied","Data":"4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8"}
Nov 03 20:10:23.350346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.350261    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eb1b497e1b5e8ff006b7e752a0f19ffbdbed1ad313d238e092e54b6ea8b52d8"
Nov 03 20:10:23.352327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.352303    5424 generic.go:356] "Generic (PLEG): container finished" podID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerID="b9eec47cadfeb9d894f4075dd880b0e5e9f6c074a9df5226295203d2a8f06112" exitCode=0
Nov 03 20:10:23.352450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.352381    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr" event={"ID":"75832162-9744-43f9-84b4-2ccbf4a4bd78","Type":"ContainerDied","Data":"b9eec47cadfeb9d894f4075dd880b0e5e9f6c074a9df5226295203d2a8f06112"}
Nov 03 20:10:23.473448 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.473416    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:10:23.501358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.501330    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\" (UniqueName: \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\") pod \"9d6a52c3-091a-4275-b219-138d180bc720\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") "
Nov 03 20:10:23.501358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.501361    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\") pod \"9d6a52c3-091a-4275-b219-138d180bc720\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") "
Nov 03 20:10:23.501557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.501417    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\") pod \"9d6a52c3-091a-4275-b219-138d180bc720\" (UID: \"9d6a52c3-091a-4275-b219-138d180bc720\") "
Nov 03 20:10:23.502000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.501933    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle" pod "9d6a52c3-091a-4275-b219-138d180bc720" (UID: "9d6a52c3-091a-4275-b219-138d180bc720"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:23.503499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.503475    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst" pod "9d6a52c3-091a-4275-b219-138d180bc720" (UID: "9d6a52c3-091a-4275-b219-138d180bc720"). InnerVolumeSpecName "kube-api-access-hzrst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:23.507261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.507239    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util" pod "9d6a52c3-091a-4275-b219-138d180bc720" (UID: "9d6a52c3-091a-4275-b219-138d180bc720"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:23.602912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.602885    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:23.602912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.602913    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrst\" (UniqueName: \"kubernetes.io/projected/9d6a52c3-091a-4275-b219-138d180bc720-kube-api-access-hzrst\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:23.603112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:23.602924    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d6a52c3-091a-4275-b219-138d180bc720-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.358260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.358229    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48"
Nov 03 20:10:24.358712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.358230    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2jp48" event={"ID":"9d6a52c3-091a-4275-b219-138d180bc720","Type":"ContainerDied","Data":"b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657"}
Nov 03 20:10:24.358712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.358377    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d815b286dac296e6955056fbba789738a9d066f410a67aa0be325582a15657"
Nov 03 20:10:24.360657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.360629    5424 generic.go:356] "Generic (PLEG): container finished" podID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerID="24d66cb67f81e99679c9c3e34148ebdd6aa1432739e9889f32e4332d7343c8a6" exitCode=0
Nov 03 20:10:24.360803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.360662    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9" event={"ID":"748b270c-46d1-450d-a8e6-44e8c4ecbcd2","Type":"ContainerDied","Data":"24d66cb67f81e99679c9c3e34148ebdd6aa1432739e9889f32e4332d7343c8a6"}
Nov 03 20:10:24.362706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.362685    5424 generic.go:356] "Generic (PLEG): container finished" podID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerID="7128d93331943ef53a6259257bd8de691e1486d2b42443eb8344e33bfaad9ca3" exitCode=0
Nov 03 20:10:24.362800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.362713    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerDied","Data":"7128d93331943ef53a6259257bd8de691e1486d2b42443eb8344e33bfaad9ca3"}
Nov 03 20:10:24.478758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.478732    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:10:24.496694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.496672    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:10:24.510860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.510843    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\" (UniqueName: \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\") pod \"75832162-9744-43f9-84b4-2ccbf4a4bd78\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") "
Nov 03 20:10:24.510927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.510918    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\") pod \"75832162-9744-43f9-84b4-2ccbf4a4bd78\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") "
Nov 03 20:10:24.510983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.510973    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\") pod \"75832162-9744-43f9-84b4-2ccbf4a4bd78\" (UID: \"75832162-9744-43f9-84b4-2ccbf4a4bd78\") "
Nov 03 20:10:24.511390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.511362    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle" pod "75832162-9744-43f9-84b4-2ccbf4a4bd78" (UID: "75832162-9744-43f9-84b4-2ccbf4a4bd78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:24.513115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.513091    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk" pod "75832162-9744-43f9-84b4-2ccbf4a4bd78" (UID: "75832162-9744-43f9-84b4-2ccbf4a4bd78"). InnerVolumeSpecName "kube-api-access-fmvjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:24.516928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.516909    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util" pod "75832162-9744-43f9-84b4-2ccbf4a4bd78" (UID: "75832162-9744-43f9-84b4-2ccbf4a4bd78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:24.612282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612216    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\") pod \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") "
Nov 03 20:10:24.612282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612256    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\" (UniqueName: \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\") pod \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") "
Nov 03 20:10:24.612460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612292    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\") pod \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\" (UID: \"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19\") "
Nov 03 20:10:24.612460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612458    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvjk\" (UniqueName: \"kubernetes.io/projected/75832162-9744-43f9-84b4-2ccbf4a4bd78-kube-api-access-fmvjk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.612590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612470    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.612590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612479    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75832162-9744-43f9-84b4-2ccbf4a4bd78-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.612859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.612833    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle" pod "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" (UID: "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:24.614490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.614464    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm" pod "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" (UID: "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19"). InnerVolumeSpecName "kube-api-access-8qvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:24.617578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.617554    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util" pod "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" (UID: "b0e85217-5ecd-4a0f-9842-daa0ed1b5c19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:24.713030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.712974    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.713030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.713022    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:24.713030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:24.713031    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvtm\" (UniqueName: \"kubernetes.io/projected/b0e85217-5ecd-4a0f-9842-daa0ed1b5c19-kube-api-access-8qvtm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.369112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.369081    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t"
Nov 03 20:10:25.369569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.369120    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ztp8t" event={"ID":"b0e85217-5ecd-4a0f-9842-daa0ed1b5c19","Type":"ContainerDied","Data":"9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da"}
Nov 03 20:10:25.369569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.369175    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b05e9926dfb7a6f9a94a928bbf915ae1466f4b41213d7f075d53cc620b013da"
Nov 03 20:10:25.371086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.371063    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr" event={"ID":"75832162-9744-43f9-84b4-2ccbf4a4bd78","Type":"ContainerDied","Data":"8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c"}
Nov 03 20:10:25.371196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.371100    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b89fe85c8114e1533d7038b3c2c40ad601409f446415e10b779dbb19847f80c"
Nov 03 20:10:25.371196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.371070    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7785jr"
Nov 03 20:10:25.496050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.496026    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:10:25.517029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.517006    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:10:25.520106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.520085    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\") pod \"c4daec27-811e-4e06-830d-5a63e15f7f66\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") "
Nov 03 20:10:25.520208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.520129    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\") pod \"c4daec27-811e-4e06-830d-5a63e15f7f66\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") "
Nov 03 20:10:25.520208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.520187    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\" (UniqueName: \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\") pod \"c4daec27-811e-4e06-830d-5a63e15f7f66\" (UID: \"c4daec27-811e-4e06-830d-5a63e15f7f66\") "
Nov 03 20:10:25.520715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.520692    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle" pod "c4daec27-811e-4e06-830d-5a63e15f7f66" (UID: "c4daec27-811e-4e06-830d-5a63e15f7f66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:25.522225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.522205    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr" pod "c4daec27-811e-4e06-830d-5a63e15f7f66" (UID: "c4daec27-811e-4e06-830d-5a63e15f7f66"). InnerVolumeSpecName "kube-api-access-gdhpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:25.525351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.525332    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util" pod "c4daec27-811e-4e06-830d-5a63e15f7f66" (UID: "c4daec27-811e-4e06-830d-5a63e15f7f66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:25.621584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621494    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\") pod \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") "
Nov 03 20:10:25.621584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621569    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\" (UniqueName: \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\") pod \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") "
Nov 03 20:10:25.621804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621611    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\") pod \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\" (UID: \"748b270c-46d1-450d-a8e6-44e8c4ecbcd2\") "
Nov 03 20:10:25.621877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621856    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdhpr\" (UniqueName: \"kubernetes.io/projected/c4daec27-811e-4e06-830d-5a63e15f7f66-kube-api-access-gdhpr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.621934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621881    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.621934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621896    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4daec27-811e-4e06-830d-5a63e15f7f66-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.622011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.621982    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle" pod "748b270c-46d1-450d-a8e6-44e8c4ecbcd2" (UID: "748b270c-46d1-450d-a8e6-44e8c4ecbcd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:25.623848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.623825    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj" pod "748b270c-46d1-450d-a8e6-44e8c4ecbcd2" (UID: "748b270c-46d1-450d-a8e6-44e8c4ecbcd2"). InnerVolumeSpecName "kube-api-access-6h2wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:25.625524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.625502    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util" pod "748b270c-46d1-450d-a8e6-44e8c4ecbcd2" (UID: "748b270c-46d1-450d-a8e6-44e8c4ecbcd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:25.722512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.722474    5424 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-bundle\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.722512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.722504    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2wj\" (UniqueName: \"kubernetes.io/projected/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-kube-api-access-6h2wj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:25.722512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:25.722513    5424 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/748b270c-46d1-450d-a8e6-44e8c4ecbcd2-util\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:10:26.147239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147204    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"]
Nov 03 20:10:26.147469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147457    5424 cpu_manager.go:414] "containerMap: removing container" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147470    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147480    5424 cpu_manager.go:414] "containerMap: removing container" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147485    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147492    5424 cpu_manager.go:414] "containerMap: removing container" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="extract"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147497    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="extract"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147506    5424 cpu_manager.go:414] "containerMap: removing container" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="util"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147510    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="util"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147517    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147521    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="pull"
Nov 03 20:10:26.147521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147547    5424 cpu_manager.go:414] "containerMap: removing container" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147553    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147562    5424 cpu_manager.go:414] "containerMap: removing container" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147569    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147577    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147583    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147590    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147610    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147621    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147625    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="pull"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147630    5424 cpu_manager.go:414] "containerMap: removing container" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147635    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147643    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147648    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147656    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147660    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147666    5424 cpu_manager.go:414] "containerMap: removing container" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147670    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147675    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147679    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147688    5424 cpu_manager.go:414] "containerMap: removing container" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147692    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147698    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147702    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="util"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147707    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147711    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147752    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6a52c3-091a-4275-b219-138d180bc720" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147759    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="75832162-9744-43f9-84b4-2ccbf4a4bd78" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147765    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e85217-5ecd-4a0f-9842-daa0ed1b5c19" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147770    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f9ed64-e961-4d13-99c7-92b76d0e0924" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147776    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="748b270c-46d1-450d-a8e6-44e8c4ecbcd2" containerName="extract"
Nov 03 20:10:26.147868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.147781    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4daec27-811e-4e06-830d-5a63e15f7f66" containerName="extract"
Nov 03 20:10:26.150011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.149995    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.169661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.169626    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 20:10:26.169887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.169870    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 20:10:26.170875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.170856    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"]
Nov 03 20:10:26.227282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.227193    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvqm\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-kube-api-access-jpvqm\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.227282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.227232    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.250268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.250243    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hlvvn\""
Nov 03 20:10:26.328114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.328081    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvqm\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-kube-api-access-jpvqm\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.328279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.328118    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.359610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.359576    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.360235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.360215    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvqm\" (UniqueName: \"kubernetes.io/projected/2cb0b7a8-707d-4413-b244-3e2f313b0e48-kube-api-access-jpvqm\") pod \"cert-manager-webhook-5dbbbf75cc-ctrtq\" (UID: \"2cb0b7a8-707d-4413-b244-3e2f313b0e48\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.377090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.377061    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9"
Nov 03 20:10:26.377466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.377059    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gxdr9" event={"ID":"748b270c-46d1-450d-a8e6-44e8c4ecbcd2","Type":"ContainerDied","Data":"e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5"}
Nov 03 20:10:26.377466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.377181    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0989c8394fddebf88b1b0cead805ea7cc323cc8d18d4294f5533ea220d783d5"
Nov 03 20:10:26.379302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.379141    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm" event={"ID":"c4daec27-811e-4e06-830d-5a63e15f7f66","Type":"ContainerDied","Data":"88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b"}
Nov 03 20:10:26.379302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.379162    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5shtm"
Nov 03 20:10:26.379302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.379188    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88dde6887e5d8ed9b14fa23fa37be79df74bf5d562aea7d66946f45fab4bdc8b"
Nov 03 20:10:26.470415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.470375    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:26.763332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:26.763308    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"]
Nov 03 20:10:26.764433 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:10:26.764409    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb0b7a8_707d_4413_b244_3e2f313b0e48.slice/crio-21cf7a06e4f54ddbd0896410c369a817c5548838ec7bf1230597f6d8c70af5fa WatchSource:0}: Error finding container 21cf7a06e4f54ddbd0896410c369a817c5548838ec7bf1230597f6d8c70af5fa: Status 404 returned error can't find the container with id 21cf7a06e4f54ddbd0896410c369a817c5548838ec7bf1230597f6d8c70af5fa
Nov 03 20:10:27.390718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.390682    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" event={"ID":"2cb0b7a8-707d-4413-b244-3e2f313b0e48","Type":"ContainerStarted","Data":"21cf7a06e4f54ddbd0896410c369a817c5548838ec7bf1230597f6d8c70af5fa"}
Nov 03 20:10:27.466731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.466688    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"]
Nov 03 20:10:27.469626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.469595    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.491547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.491506    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:10:27.493788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.493765    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:10:27.494077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.494058    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:10:27.494077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.494060    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rsm5h\""
Nov 03 20:10:27.497277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.497242    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:10:27.497421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.497279    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:10:27.536312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536221    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"]
Nov 03 20:10:27.536479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536438    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/981567a1-a8b5-4912-866c-47a06b2e2a35-kube-api-access-ptqdk\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.536635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536495    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.536635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536623    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.536746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536657    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.536746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536688    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.536856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.536812    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637340    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637403    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637436    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637462    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637489    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.637594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.637518    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/981567a1-a8b5-4912-866c-47a06b2e2a35-kube-api-access-ptqdk\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.638219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.638193    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.641045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.640962    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.642046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.641983    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981567a1-a8b5-4912-866c-47a06b2e2a35-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.642707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.642685    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.643942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.643915    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981567a1-a8b5-4912-866c-47a06b2e2a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.670784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.670723    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqdk\" (UniqueName: \"kubernetes.io/projected/981567a1-a8b5-4912-866c-47a06b2e2a35-kube-api-access-ptqdk\") pod \"opendatahub-operator-controller-manager-6fd567655f-nscg8\" (UID: \"981567a1-a8b5-4912-866c-47a06b2e2a35\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.786202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.786165    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:27.893978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.893893    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"]
Nov 03 20:10:27.896279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.896256    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:27.915605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.915563    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nmpvf\""
Nov 03 20:10:27.920154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.920122    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"]
Nov 03 20:10:27.939407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.939189    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgwg\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-kube-api-access-2sgwg\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:27.939407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:27.939315    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.040424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.040387    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgwg\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-kube-api-access-2sgwg\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.040625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.040444    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.061922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.061859    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.062625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.062522    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgwg\" (UniqueName: \"kubernetes.io/projected/584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb-kube-api-access-2sgwg\") pod \"cert-manager-cainjector-5c57bbd4fc-t9qfx\" (UID: \"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.136089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.136057    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"]
Nov 03 20:10:28.138141 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:10:28.138109    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981567a1_a8b5_4912_866c_47a06b2e2a35.slice/crio-a1bfbcbe037cbab31578102df643b69f3e53c0f4a2e55751bc4efa1e5e46c72b WatchSource:0}: Error finding container a1bfbcbe037cbab31578102df643b69f3e53c0f4a2e55751bc4efa1e5e46c72b: Status 404 returned error can't find the container with id a1bfbcbe037cbab31578102df643b69f3e53c0f4a2e55751bc4efa1e5e46c72b
Nov 03 20:10:28.211737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.211651    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"
Nov 03 20:10:28.398952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.398901    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8" event={"ID":"981567a1-a8b5-4912-866c-47a06b2e2a35","Type":"ContainerStarted","Data":"a1bfbcbe037cbab31578102df643b69f3e53c0f4a2e55751bc4efa1e5e46c72b"}
Nov 03 20:10:28.519107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:28.519003    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx"]
Nov 03 20:10:28.521403 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:10:28.521374    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod584cc6b0_f8d4_4c11_a0b5_3cee99d6b1bb.slice/crio-937db5cb0009f2dcbcaf930973ddfcfe51d25c96fed3053f43c2085ab443335d WatchSource:0}: Error finding container 937db5cb0009f2dcbcaf930973ddfcfe51d25c96fed3053f43c2085ab443335d: Status 404 returned error can't find the container with id 937db5cb0009f2dcbcaf930973ddfcfe51d25c96fed3053f43c2085ab443335d
Nov 03 20:10:29.407128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:29.407047    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx" event={"ID":"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb","Type":"ContainerStarted","Data":"937db5cb0009f2dcbcaf930973ddfcfe51d25c96fed3053f43c2085ab443335d"}
Nov 03 20:10:31.420280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:31.420237    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" event={"ID":"2cb0b7a8-707d-4413-b244-3e2f313b0e48","Type":"ContainerStarted","Data":"28c154757ca6984b4d09e3a1b91ec273a4609609fe1492c07cb705712677e40b"}
Nov 03 20:10:31.420755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:31.420389    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:31.422369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:31.422330    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx" event={"ID":"584cc6b0-f8d4-4c11-a0b5-3cee99d6b1bb","Type":"ContainerStarted","Data":"af7e91a2cf14ec5ce315dcb8d0c482486a045820f21f136fef91931c00136ecf"}
Nov 03 20:10:31.465897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:31.465847    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" podStartSLOduration=1.5393575689999999 podStartE2EDuration="5.465826094s" totalImagesPullingTime="3.926468525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:31.465239411 +0000 UTC m=+373.674255407" watchObservedRunningTime="2026-11-03 20:10:31.465826094 +0000 UTC m=+373.674842092"
Nov 03 20:10:31.504299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:31.504238    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-t9qfx" podStartSLOduration=2.284252057 podStartE2EDuration="4.504222875s" totalImagesPullingTime="2.219970818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:31.503092107 +0000 UTC m=+373.712108104" watchObservedRunningTime="2026-11-03 20:10:31.504222875 +0000 UTC m=+373.713238871"
Nov 03 20:10:32.428276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:32.428189    5424 generic.go:356] "Generic (PLEG): container finished" podID="981567a1-a8b5-4912-866c-47a06b2e2a35" containerID="2b409c12fb83f3cc9850bb5ec08310d7d113b80cb8cea2b6aeb9a78c66381c88" exitCode=0
Nov 03 20:10:32.428868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:32.428267    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8" event={"ID":"981567a1-a8b5-4912-866c-47a06b2e2a35","Type":"ContainerDied","Data":"2b409c12fb83f3cc9850bb5ec08310d7d113b80cb8cea2b6aeb9a78c66381c88"}
Nov 03 20:10:33.435323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:33.435284    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8" event={"ID":"981567a1-a8b5-4912-866c-47a06b2e2a35","Type":"ContainerStarted","Data":"22dfe2aa33400d31ccaacc894b632c899919c0f343d6ade9956644424493524f"}
Nov 03 20:10:33.435852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:33.435384    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:33.591480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:33.591431    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8" podStartSLOduration=2.359561824 podStartE2EDuration="6.591417709s" totalImagesPullingTime="3.813559053s" totalInitContainerRuntime="418.296832ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:33.578655692 +0000 UTC m=+375.787671688" watchObservedRunningTime="2026-11-03 20:10:33.591417709 +0000 UTC m=+375.800433706"
Nov 03 20:10:35.305716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.305676    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hpz7m"]
Nov 03 20:10:35.307874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.307857    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.335107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.335074    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nxwbk\""
Nov 03 20:10:35.336182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.336157    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hpz7m"]
Nov 03 20:10:35.409519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.409481    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-bound-sa-token\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.409708 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.409553    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf5g\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-kube-api-access-5pf5g\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.510277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.510236    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-bound-sa-token\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.510451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.510299    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf5g\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-kube-api-access-5pf5g\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.537994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.537964    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf5g\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-kube-api-access-5pf5g\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.537994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.537975    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1492a022-256b-46cd-bb9a-3188ef1e7229-bound-sa-token\") pod \"cert-manager-597b95cb84-hpz7m\" (UID: \"1492a022-256b-46cd-bb9a-3188ef1e7229\") " pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.618261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.618179    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hpz7m"
Nov 03 20:10:35.908773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:35.908744    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hpz7m"]
Nov 03 20:10:35.909850 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:10:35.909826    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1492a022_256b_46cd_bb9a_3188ef1e7229.slice/crio-20b7782247ed3365ceec5fa28a64487545a29ec829a6da94d2d63c66cc8914f0 WatchSource:0}: Error finding container 20b7782247ed3365ceec5fa28a64487545a29ec829a6da94d2d63c66cc8914f0: Status 404 returned error can't find the container with id 20b7782247ed3365ceec5fa28a64487545a29ec829a6da94d2d63c66cc8914f0
Nov 03 20:10:36.450367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:36.450332    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hpz7m" event={"ID":"1492a022-256b-46cd-bb9a-3188ef1e7229","Type":"ContainerStarted","Data":"dcdb53e4c2c36487ac39c6d1dc4eeeb6de46fddff3da950f605c31b25592556e"}
Nov 03 20:10:36.450367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:36.450380    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hpz7m" event={"ID":"1492a022-256b-46cd-bb9a-3188ef1e7229","Type":"ContainerStarted","Data":"20b7782247ed3365ceec5fa28a64487545a29ec829a6da94d2d63c66cc8914f0"}
Nov 03 20:10:36.490849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:36.490791    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hpz7m" podStartSLOduration=1.490777748 podStartE2EDuration="1.490777748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:36.489707678 +0000 UTC m=+378.698723675" watchObservedRunningTime="2026-11-03 20:10:36.490777748 +0000 UTC m=+378.699793744"
Nov 03 20:10:37.430585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:37.430556    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq"
Nov 03 20:10:42.285860 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:10:42.285821    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:10:43.771240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.771206    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"]
Nov 03 20:10:43.775236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.775219    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:43.803315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.803288    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7lj4v\""
Nov 03 20:10:43.803315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.803317    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:10:43.803597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.803396    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 20:10:43.804612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.804591    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"]
Nov 03 20:10:43.880155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.880121    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-tmp\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:43.880369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.880346    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzc5p\" (UniqueName: \"kubernetes.io/projected/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-kube-api-access-rzc5p\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:43.981221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.981174    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-tmp\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:43.981221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.981224    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzc5p\" (UniqueName: \"kubernetes.io/projected/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-kube-api-access-rzc5p\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:43.981674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:43.981648    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-tmp\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:44.001440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:44.001418    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzc5p\" (UniqueName: \"kubernetes.io/projected/0799ae4b-41b8-4c12-b2cc-cb7a26e03986-kube-api-access-rzc5p\") pod \"jobset-operator-674cd59dc5-xb5dj\" (UID: \"0799ae4b-41b8-4c12-b2cc-cb7a26e03986\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:44.086267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:44.086176    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"
Nov 03 20:10:44.390490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:44.390413    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj"]
Nov 03 20:10:44.393830 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:10:44.393802    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0799ae4b_41b8_4c12_b2cc_cb7a26e03986.slice/crio-980882d7803a3dd275dfd1035368cf8aa154182f5867b8888ae99b13982690a7 WatchSource:0}: Error finding container 980882d7803a3dd275dfd1035368cf8aa154182f5867b8888ae99b13982690a7: Status 404 returned error can't find the container with id 980882d7803a3dd275dfd1035368cf8aa154182f5867b8888ae99b13982690a7
Nov 03 20:10:44.444145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:44.444117    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8"
Nov 03 20:10:44.485580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:44.485540    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj" event={"ID":"0799ae4b-41b8-4c12-b2cc-cb7a26e03986","Type":"ContainerStarted","Data":"980882d7803a3dd275dfd1035368cf8aa154182f5867b8888ae99b13982690a7"}
Nov 03 20:10:47.500072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:47.500038    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj" event={"ID":"0799ae4b-41b8-4c12-b2cc-cb7a26e03986","Type":"ContainerStarted","Data":"54a0eea3c3accef156e0f3ec53181d73b9523a2968fe18d162a48998f2053b3c"}
Nov 03 20:10:47.542766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:47.542712    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-xb5dj" podStartSLOduration=1.900252791 podStartE2EDuration="4.542697448s" totalImagesPullingTime="2.642444657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:47.5421322 +0000 UTC m=+389.751148194" watchObservedRunningTime="2026-11-03 20:10:47.542697448 +0000 UTC m=+389.751713442"
Nov 03 20:10:59.971581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:59.971523    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"]
Nov 03 20:10:59.976573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:59.976520    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:10:59.996862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:10:59.996834    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"]
Nov 03 20:11:00.008048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.008016    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:11:00.008301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.008095    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 20:11:00.008411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.008100    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 20:11:00.008411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.008142    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6qpp4\""
Nov 03 20:11:00.008411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.008161    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:11:00.123062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.123024    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a807a22-e50c-4464-9b68-4ec5d33f499f-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.123235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.123104    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7jc\" (UniqueName: \"kubernetes.io/projected/3a807a22-e50c-4464-9b68-4ec5d33f499f-kube-api-access-zw7jc\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.123235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.123178    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a807a22-e50c-4464-9b68-4ec5d33f499f-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.224132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.224050    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7jc\" (UniqueName: \"kubernetes.io/projected/3a807a22-e50c-4464-9b68-4ec5d33f499f-kube-api-access-zw7jc\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.224132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.224086    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a807a22-e50c-4464-9b68-4ec5d33f499f-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.224316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.224135    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a807a22-e50c-4464-9b68-4ec5d33f499f-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.226642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.226610    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a807a22-e50c-4464-9b68-4ec5d33f499f-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.226892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.226869    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a807a22-e50c-4464-9b68-4ec5d33f499f-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.252549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.252507    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7jc\" (UniqueName: \"kubernetes.io/projected/3a807a22-e50c-4464-9b68-4ec5d33f499f-kube-api-access-zw7jc\") pod \"kube-auth-proxy-6fd6dbbcd5-m28cr\" (UID: \"3a807a22-e50c-4464-9b68-4ec5d33f499f\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.288233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.288203    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"
Nov 03 20:11:00.581433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:00.581404    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr"]
Nov 03 20:11:00.582075 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:11:00.582042    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a807a22_e50c_4464_9b68_4ec5d33f499f.slice/crio-eb1ec212e5116715f2bdce7911214edd9841155bcd9467b0346316c33180c3b3 WatchSource:0}: Error finding container eb1ec212e5116715f2bdce7911214edd9841155bcd9467b0346316c33180c3b3: Status 404 returned error can't find the container with id eb1ec212e5116715f2bdce7911214edd9841155bcd9467b0346316c33180c3b3
Nov 03 20:11:01.560352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:01.560311    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr" event={"ID":"3a807a22-e50c-4464-9b68-4ec5d33f499f","Type":"ContainerStarted","Data":"eb1ec212e5116715f2bdce7911214edd9841155bcd9467b0346316c33180c3b3"}
Nov 03 20:11:03.571203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:03.571169    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr" event={"ID":"3a807a22-e50c-4464-9b68-4ec5d33f499f","Type":"ContainerStarted","Data":"6ef440e6c463d7531862f45cf29936a487b3f6096ca0a0d1cc10497aec2a8e0d"}
Nov 03 20:11:03.641856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:03.641804    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-m28cr" podStartSLOduration=2.221851011 podStartE2EDuration="4.641789871s" totalImagesPullingTime="2.41993886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:03.641474446 +0000 UTC m=+405.850490444" watchObservedRunningTime="2026-11-03 20:11:03.641789871 +0000 UTC m=+405.850805892"
Nov 03 20:11:04.999246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:04.999206    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"]
Nov 03 20:11:05.014950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.014924    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.022305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.022171    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"]
Nov 03 20:11:05.032905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.032873    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 20:11:05.033056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.033039    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4cqpf\""
Nov 03 20:11:05.042166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.042145    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 20:11:05.042166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.042152    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 20:11:05.165350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.165311    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbf45200-f39f-48a1-b135-62e08729229c-manager-config\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.165509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.165380    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-cert\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.165509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.165465    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhmn\" (UniqueName: \"kubernetes.io/projected/bbf45200-f39f-48a1-b135-62e08729229c-kube-api-access-vzhmn\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.165624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.165510    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-metrics-certs\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.266508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.266422    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhmn\" (UniqueName: \"kubernetes.io/projected/bbf45200-f39f-48a1-b135-62e08729229c-kube-api-access-vzhmn\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.266508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.266475    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-metrics-certs\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.266771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.266591    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbf45200-f39f-48a1-b135-62e08729229c-manager-config\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.266771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.266688    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-cert\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.267173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.267153    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bbf45200-f39f-48a1-b135-62e08729229c-manager-config\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.269175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.269156    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-metrics-certs\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.269273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.269245    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbf45200-f39f-48a1-b135-62e08729229c-cert\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.286864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.286833    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhmn\" (UniqueName: \"kubernetes.io/projected/bbf45200-f39f-48a1-b135-62e08729229c-kube-api-access-vzhmn\") pod \"jobset-controller-manager-764545c97f-kx7db\" (UID: \"bbf45200-f39f-48a1-b135-62e08729229c\") " pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.333600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.333567    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:05.655203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:05.655173    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"]
Nov 03 20:11:05.655691 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:11:05.655667    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf45200_f39f_48a1_b135_62e08729229c.slice/crio-8e3470e2c8ed0dacef6fe2225e67ce8d3e42157107f2bdd47b322207688e9e4c WatchSource:0}: Error finding container 8e3470e2c8ed0dacef6fe2225e67ce8d3e42157107f2bdd47b322207688e9e4c: Status 404 returned error can't find the container with id 8e3470e2c8ed0dacef6fe2225e67ce8d3e42157107f2bdd47b322207688e9e4c
Nov 03 20:11:06.586941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:06.586901    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db" event={"ID":"bbf45200-f39f-48a1-b135-62e08729229c","Type":"ContainerStarted","Data":"8e3470e2c8ed0dacef6fe2225e67ce8d3e42157107f2bdd47b322207688e9e4c"}
Nov 03 20:11:08.598004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:08.597971    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db" event={"ID":"bbf45200-f39f-48a1-b135-62e08729229c","Type":"ContainerStarted","Data":"2de8dfdec43cf55540d3c6e1a6888c1db84b312028bc6af56ceff080cb3f3b03"}
Nov 03 20:11:08.598436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:08.598048    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:08.638710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:08.638654    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db" podStartSLOduration=2.114243005 podStartE2EDuration="4.638640567s" totalImagesPullingTime="2.524397562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:08.637040725 +0000 UTC m=+410.846056722" watchObservedRunningTime="2026-11-03 20:11:08.638640567 +0000 UTC m=+410.847656563"
Nov 03 20:11:12.285563 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:11:12.285505    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:11:19.608190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:11:19.608156    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-764545c97f-kx7db"
Nov 03 20:11:42.284989 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:11:42.284953    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:12:03.741376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.741291    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:12:03.743998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.743975    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.769661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.769623    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:12:03.771383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.771353    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:12:03.778747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.778726    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:12:03.820405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.820377    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 20:12:03.820669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.820652    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5b4v7\""
Nov 03 20:12:03.854444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.854414    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.854636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.854479    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxtl\" (UniqueName: \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.955149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.955109    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.955329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.955211    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxtl\" (UniqueName: \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.957973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.957944    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:03.974792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:03.974763    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxtl\" (UniqueName: \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\") pod \"llmisvc-controller-manager-869d5dcc5d-7kgz6\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:04.055738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:04.055654    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:04.346501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:04.346474    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:12:04.347993 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:12:04.347968    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podae8af4b7_c49d_4836_988d_a767069d4c42.slice/crio-9cedd9c82c2d1475c89f0b436aa32fe509a0f42ef7059a11bbaa9bc1059ffdbe WatchSource:0}: Error finding container 9cedd9c82c2d1475c89f0b436aa32fe509a0f42ef7059a11bbaa9bc1059ffdbe: Status 404 returned error can't find the container with id 9cedd9c82c2d1475c89f0b436aa32fe509a0f42ef7059a11bbaa9bc1059ffdbe
Nov 03 20:12:04.838008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:04.837969    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" event={"ID":"ae8af4b7-c49d-4836-988d-a767069d4c42","Type":"ContainerStarted","Data":"9cedd9c82c2d1475c89f0b436aa32fe509a0f42ef7059a11bbaa9bc1059ffdbe"}
Nov 03 20:12:06.848449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:06.848414    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" event={"ID":"ae8af4b7-c49d-4836-988d-a767069d4c42","Type":"ContainerStarted","Data":"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"}
Nov 03 20:12:06.848924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:06.848479    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:06.887277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:06.887223    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" podStartSLOduration=1.798321123 podStartE2EDuration="3.887208462s" totalImagesPullingTime="2.088887339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:06.886426738 +0000 UTC m=+469.095442746" watchObservedRunningTime="2026-11-03 20:12:06.887208462 +0000 UTC m=+469.096224501"
Nov 03 20:12:12.286812 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:12:12.286773    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:12:37.855000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:12:37.854971    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:12:42.284372 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:12:42.284334    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:13:08.159606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.159576    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"]
Nov 03 20:13:08.162089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.162069    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.179065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.179032    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"]
Nov 03 20:13:08.190854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.190825    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79d945a9-0182-4a44-9280-92c12cbaca8f-cert\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.191003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.190889    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q2q\" (UniqueName: \"kubernetes.io/projected/79d945a9-0182-4a44-9280-92c12cbaca8f-kube-api-access-55q2q\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.291915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.291877    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79d945a9-0182-4a44-9280-92c12cbaca8f-cert\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.292113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.291944    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q2q\" (UniqueName: \"kubernetes.io/projected/79d945a9-0182-4a44-9280-92c12cbaca8f-kube-api-access-55q2q\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.294403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.294382    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79d945a9-0182-4a44-9280-92c12cbaca8f-cert\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.311594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.311566    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q2q\" (UniqueName: \"kubernetes.io/projected/79d945a9-0182-4a44-9280-92c12cbaca8f-kube-api-access-55q2q\") pod \"llmisvc-controller-manager-6466c554d8-s8phc\" (UID: \"79d945a9-0182-4a44-9280-92c12cbaca8f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.474097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.474007    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:08.765747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:08.765721    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"]
Nov 03 20:13:08.766395 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:13:08.766373    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d945a9_0182_4a44_9280_92c12cbaca8f.slice/crio-8dcbd31ea980e2318441b418aacc56267b2fc6c30db394483281b4edf1cb09e6 WatchSource:0}: Error finding container 8dcbd31ea980e2318441b418aacc56267b2fc6c30db394483281b4edf1cb09e6: Status 404 returned error can't find the container with id 8dcbd31ea980e2318441b418aacc56267b2fc6c30db394483281b4edf1cb09e6
Nov 03 20:13:09.116656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:09.116615    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc" event={"ID":"79d945a9-0182-4a44-9280-92c12cbaca8f","Type":"ContainerStarted","Data":"f366596ad9b6e7f7de3d353c494ff755c8c2bc3ec1e0c4a09b291e85eca64682"}
Nov 03 20:13:09.116813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:09.116679    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc" event={"ID":"79d945a9-0182-4a44-9280-92c12cbaca8f","Type":"ContainerStarted","Data":"8dcbd31ea980e2318441b418aacc56267b2fc6c30db394483281b4edf1cb09e6"}
Nov 03 20:13:09.116813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:09.116719    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:09.155639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:09.155574    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc" podStartSLOduration=0.878849141 podStartE2EDuration="1.15555861s" totalImagesPullingTime="276.709469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:09.154790845 +0000 UTC m=+531.363806841" watchObservedRunningTime="2026-11-03 20:13:09.15555861 +0000 UTC m=+531.364574601"
Nov 03 20:13:12.285076 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:13:12.284964    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:13:40.123962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.123882    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s8phc"
Nov 03 20:13:40.187593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.187566    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:13:40.187783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.187759    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" podUID="ae8af4b7-c49d-4836-988d-a767069d4c42" containerName="manager" containerID="cri-o://13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016" gracePeriod=30
Nov 03 20:13:40.430111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.430086    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:13:40.564957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.564922    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\" (UniqueName: \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\") pod \"ae8af4b7-c49d-4836-988d-a767069d4c42\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") "
Nov 03 20:13:40.565128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.564966    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\" (UniqueName: \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\") pod \"ae8af4b7-c49d-4836-988d-a767069d4c42\" (UID: \"ae8af4b7-c49d-4836-988d-a767069d4c42\") "
Nov 03 20:13:40.567372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.567342    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert" pod "ae8af4b7-c49d-4836-988d-a767069d4c42" (UID: "ae8af4b7-c49d-4836-988d-a767069d4c42"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:40.567481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.567389    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl" pod "ae8af4b7-c49d-4836-988d-a767069d4c42" (UID: "ae8af4b7-c49d-4836-988d-a767069d4c42"). InnerVolumeSpecName "kube-api-access-sjxtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:40.666068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.665978    5424 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae8af4b7-c49d-4836-988d-a767069d4c42-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:13:40.666068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:40.666010    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxtl\" (UniqueName: \"kubernetes.io/projected/ae8af4b7-c49d-4836-988d-a767069d4c42-kube-api-access-sjxtl\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:13:41.258167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.258133    5424 generic.go:356] "Generic (PLEG): container finished" podID="ae8af4b7-c49d-4836-988d-a767069d4c42" containerID="13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016" exitCode=0
Nov 03 20:13:41.258584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.258186    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"
Nov 03 20:13:41.258584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.258217    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" event={"ID":"ae8af4b7-c49d-4836-988d-a767069d4c42","Type":"ContainerDied","Data":"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"}
Nov 03 20:13:41.258584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.258267    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6" event={"ID":"ae8af4b7-c49d-4836-988d-a767069d4c42","Type":"ContainerDied","Data":"9cedd9c82c2d1475c89f0b436aa32fe509a0f42ef7059a11bbaa9bc1059ffdbe"}
Nov 03 20:13:41.258584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.258291    5424 scope.go:122] "RemoveContainer" containerID="13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"
Nov 03 20:13:41.267713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.267698    5424 scope.go:122] "RemoveContainer" containerID="13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"
Nov 03 20:13:41.268009 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:13:41.267989    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016\": container with ID starting with 13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016 not found: ID does not exist" containerID="13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"
Nov 03 20:13:41.268067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.268021    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016"} err="failed to get container status \"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016\": rpc error: code = NotFound desc = could not find container \"13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016\": container with ID starting with 13ad750262c44d7d4c9f243dd267b30d6c44c588201d5f4987e7e4f61c01c016 not found: ID does not exist"
Nov 03 20:13:41.309166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.309127    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:13:41.321462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.321432    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7kgz6"]
Nov 03 20:13:41.918056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:13:41.918024    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8af4b7-c49d-4836-988d-a767069d4c42" path="/var/lib/kubelet/pods/ae8af4b7-c49d-4836-988d-a767069d4c42/volumes"
Nov 03 20:13:42.284903 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:13:42.284867    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:14:12.285349 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:14:12.285315    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:14:27.031271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.031234    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"]
Nov 03 20:14:27.031702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.031622    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ae8af4b7-c49d-4836-988d-a767069d4c42" containerName="manager"
Nov 03 20:14:27.031702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.031634    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8af4b7-c49d-4836-988d-a767069d4c42" containerName="manager"
Nov 03 20:14:27.031702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.031700    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8af4b7-c49d-4836-988d-a767069d4c42" containerName="manager"
Nov 03 20:14:27.034506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.034486    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.052417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.052389    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:14:27.053932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.053909    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"]
Nov 03 20:14:27.134167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.134134    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tzlpr\""
Nov 03 20:14:27.136476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136452    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136491    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136517    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136601    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136713    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136786    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4h2d\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-kube-api-access-v4h2d\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136843    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136872    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.136938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.136899    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.237876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.237837    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.237876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.237879    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.237895    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238012    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238056    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4h2d\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-kube-api-access-v4h2d\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238112    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238140    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238159    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238197    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238259    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238312    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238509    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238773    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.238870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.238817    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.240654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.240632    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.240889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.240869    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.257141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.257119    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.257427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.257410    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4h2d\" (UniqueName: \"kubernetes.io/projected/3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9-kube-api-access-v4h2d\") pod \"maas-default-gateway-openshift-default-694db55c87-mkxl5\" (UID: \"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.348757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.348683    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:27.651681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:27.651656    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"]
Nov 03 20:14:27.652487 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:14:27.652458    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3637dc05_a96f_4dae_8ae4_dcd42dfd9ac9.slice/crio-3fc80d17eedba9d5bcf5eb1b16b7c0bb3db45660b78d692d958355fdae6dd6a9 WatchSource:0}: Error finding container 3fc80d17eedba9d5bcf5eb1b16b7c0bb3db45660b78d692d958355fdae6dd6a9: Status 404 returned error can't find the container with id 3fc80d17eedba9d5bcf5eb1b16b7c0bb3db45660b78d692d958355fdae6dd6a9
Nov 03 20:14:28.465564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:28.465518    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" event={"ID":"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9","Type":"ContainerStarted","Data":"3fc80d17eedba9d5bcf5eb1b16b7c0bb3db45660b78d692d958355fdae6dd6a9"}
Nov 03 20:14:30.685440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:30.685408    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:30.685789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:30.685477    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:30.685789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:30.685506    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:31.481986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:31.481948    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" event={"ID":"3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9","Type":"ContainerStarted","Data":"fc09847722564effd93029cb7c3bd697570b8f3cb339e454d8c2e89967a01c4c"}
Nov 03 20:14:31.544319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:31.544246    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podStartSLOduration=1.514256149 podStartE2EDuration="4.544227655s" totalImagesPullingTime="3.029971506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:31.54208818 +0000 UTC m=+613.751104203" watchObservedRunningTime="2026-11-03 20:14:31.544227655 +0000 UTC m=+613.753243653"
Nov 03 20:14:32.349846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:32.349813    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:32.351143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:32.351117    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:32.351274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:32.351168    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:33.350189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:33.350146    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:33.350639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:33.350212    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:34.350071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:34.350028    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:34.350256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:34.350101    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:35.350092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:35.350061    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:35.350291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:35.350129    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:36.349518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:36.349481    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:36.349764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:36.349569    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:37.349432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:37.349395    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 20:14:37.349979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:37.349466    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 20:14:39.349717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:39.349660    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:39.350150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:39.349733    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:40.391742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:40.391693    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:40.392143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:40.391752    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:40.843440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:40.843401    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:40.843812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:40.843789    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:40.845041 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:40.845019    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5"
Nov 03 20:14:42.073801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.073772    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"]
Nov 03 20:14:42.076222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.076206    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.100462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.100435    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"]
Nov 03 20:14:42.193722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.193670    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.193722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.193721    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.193987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.193798    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9cm\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-kube-api-access-hd9cm\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.193987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.193839    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.193987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.193967    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.194117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.194022    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.194117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.194085    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.194185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.194125    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.194185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.194145    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.284841 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:14:42.284796    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:14:42.295245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295219    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295262    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295284    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295307    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295326    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295358    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295376    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295398    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9cm\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-kube-api-access-hd9cm\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295425    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295775    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295829    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.295901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295862    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.296003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.295938    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.296059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.296025    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.298006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.297983    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.298100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.298029    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.314791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.314761    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.314915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.314845    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9cm\" (UniqueName: \"kubernetes.io/projected/064e0bc1-2bbe-4ba9-8be6-1d0074655e00-kube-api-access-hd9cm\") pod \"maas-default-gateway-openshift-default-694db55c87-pd9ml\" (UID: \"064e0bc1-2bbe-4ba9-8be6-1d0074655e00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.389055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.389023    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:42.694741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.694717    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"]
Nov 03 20:14:42.696282 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:14:42.696250    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064e0bc1_2bbe_4ba9_8be6_1d0074655e00.slice/crio-6dacd8405aefbb5a5cf769107eb36a48ecbdc27ab985839b1427878d13fa6cf9 WatchSource:0}: Error finding container 6dacd8405aefbb5a5cf769107eb36a48ecbdc27ab985839b1427878d13fa6cf9: Status 404 returned error can't find the container with id 6dacd8405aefbb5a5cf769107eb36a48ecbdc27ab985839b1427878d13fa6cf9
Nov 03 20:14:42.699399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.699362    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:42.699492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.699435    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:42.699492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:42.699462    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:43.548048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:43.548012    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" event={"ID":"064e0bc1-2bbe-4ba9-8be6-1d0074655e00","Type":"ContainerStarted","Data":"662684834eb60969494d4f45348c7842d73723fc59392d11befb092d03f42c81"}
Nov 03 20:14:43.548428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:43.548073    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" event={"ID":"064e0bc1-2bbe-4ba9-8be6-1d0074655e00","Type":"ContainerStarted","Data":"6dacd8405aefbb5a5cf769107eb36a48ecbdc27ab985839b1427878d13fa6cf9"}
Nov 03 20:14:43.614210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:43.614154    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podStartSLOduration=1.61414101 podStartE2EDuration="1.61414101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:43.601239776 +0000 UTC m=+625.810255772" watchObservedRunningTime="2026-11-03 20:14:43.61414101 +0000 UTC m=+625.823157006"
Nov 03 20:14:44.389417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:44.389386    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:44.390935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:44.390908    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:44.391056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:44.390954    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:45.389894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:45.389859    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:45.390283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:45.389911    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:46.389900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:46.389863    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:46.390278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:46.389917    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:47.389936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:47.389889    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:47.390347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:47.389957    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:48.389773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:48.389742    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:48.389957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:48.389795    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:49.389445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:49.389410    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:49.389647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:49.389466    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 20:14:50.781089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:50.781047    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:50.781711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:50.781571    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:14:50.782624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:14:50.782603    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml"
Nov 03 20:15:12.285283 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:15:12.285204    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:15:25.739127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.739089    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:15:25.744686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.744651    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.764396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.764347    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:15:25.773696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.773654    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-482722-openshift-default-dockercfg-6d2xm\""
Nov 03 20:15:25.909170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909131    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909190    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909253    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909355    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909435    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzls\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909496    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909521    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909636    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:25.909745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:25.909724    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011003    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzls\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011053    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011166    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011211    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011255    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011296    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011335    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011363    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011406    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011628    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011705    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011734    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.011818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.011768    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.012074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.012052    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.013947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.013919    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.014080 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.014050    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.031771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.031741    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.032156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.032130    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzls\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-6dcdg\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.058136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.058096    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:26.401095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.401063    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:15:26.401814 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:26.401791    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ca4640_8625_4d65_97cd_5742801f27c0.slice/crio-145ffc3025412c165f2247a95a93c6a696e95a50d4de7afea09cc4418eb87fc2 WatchSource:0}: Error finding container 145ffc3025412c165f2247a95a93c6a696e95a50d4de7afea09cc4418eb87fc2: Status 404 returned error can't find the container with id 145ffc3025412c165f2247a95a93c6a696e95a50d4de7afea09cc4418eb87fc2
Nov 03 20:15:26.403964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.403929    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:26.404050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.403999    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:26.404050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.404028    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:26.747678 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.747637    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerStarted","Data":"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"}
Nov 03 20:15:26.748064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.747737    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerStarted","Data":"145ffc3025412c165f2247a95a93c6a696e95a50d4de7afea09cc4418eb87fc2"}
Nov 03 20:15:26.792249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:26.792196    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podStartSLOduration=1.792184432 podStartE2EDuration="1.792184432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.791491581 +0000 UTC m=+669.000507578" watchObservedRunningTime="2026-11-03 20:15:26.792184432 +0000 UTC m=+669.001200482"
Nov 03 20:15:27.059249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:27.059146    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:27.064440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:27.064407    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:27.752636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:27.752596    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:27.753750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:27.753731    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:36.417483 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.417448    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:15:36.419754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.419736    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.439727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.439703    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:15:36.450382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.450361    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e80203-dockercfg-z4nsg\""
Nov 03 20:15:36.472096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.472070    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e80203\""
Nov 03 20:15:36.505787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.505748    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjh9h\" (UniqueName: \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.505941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.505814    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.606321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.606288    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjh9h\" (UniqueName: \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.606500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.606335    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.607105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.607084    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.625762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.625731    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjh9h\" (UniqueName: \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\") pod \"payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:36.732210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:36.732182    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:37.050091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:37.050056    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:15:37.050839 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:37.050811    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7454d1f9_4e84_42d5_94d8_3b1dd9960841.slice/crio-f881e3df1b70423db629981be7c24de4534351e5f740417648606a0362900d00 WatchSource:0}: Error finding container f881e3df1b70423db629981be7c24de4534351e5f740417648606a0362900d00: Status 404 returned error can't find the container with id f881e3df1b70423db629981be7c24de4534351e5f740417648606a0362900d00
Nov 03 20:15:37.052742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:37.052725    5424 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:15:37.803582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:37.803478    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" event={"ID":"7454d1f9-4e84-42d5-94d8-3b1dd9960841","Type":"ContainerStarted","Data":"f881e3df1b70423db629981be7c24de4534351e5f740417648606a0362900d00"}
Nov 03 20:15:38.064096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.064005    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:15:38.066747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.066726    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.096803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.096767    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:15:38.116794 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.116754    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.116980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.116807    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp47q\" (UniqueName: \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.122063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.121856    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6fc748\""
Nov 03 20:15:38.122063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.122028    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6fc748-dockercfg-86btt\""
Nov 03 20:15:38.243557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.241745    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.243557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.241801    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp47q\" (UniqueName: \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.243557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.242922    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.261856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.261824    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp47q\" (UniqueName: \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\") pod \"payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.381163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.381074    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:38.738885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.717897    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:15:38.745506 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:38.745336    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d65d9ce_28f5_43c4_a7e7_a06ffef1a1b8.slice/crio-dbda2ab3c355e6f7a497ecdaff4de059749c5fcf96dabca7830e77cc64cd23a1 WatchSource:0}: Error finding container dbda2ab3c355e6f7a497ecdaff4de059749c5fcf96dabca7830e77cc64cd23a1: Status 404 returned error can't find the container with id dbda2ab3c355e6f7a497ecdaff4de059749c5fcf96dabca7830e77cc64cd23a1
Nov 03 20:15:38.810326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.810285    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" event={"ID":"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8","Type":"ContainerStarted","Data":"dbda2ab3c355e6f7a497ecdaff4de059749c5fcf96dabca7830e77cc64cd23a1"}
Nov 03 20:15:38.946577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.946513    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:15:38.953302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.953267    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:38.976231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.976181    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e40816\""
Nov 03 20:15:38.976231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.976199    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e40816-dockercfg-zbq6q\""
Nov 03 20:15:38.976966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:38.976946    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:15:39.052926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.052756    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.052926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.052797    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chknw\" (UniqueName: \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.154836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.153956    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.154836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.154023    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chknw\" (UniqueName: \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.154836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.154785    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.174687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.174623    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chknw\" (UniqueName: \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\") pod \"payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.267302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.267262    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:39.648665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.644354    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:15:39.660046 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:39.660001    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3968f1ea_e378_4dda_bbb2_d83835be0d92.slice/crio-4e7af37055906f5f9b93e714664dd4940bb74b0893a7da9cdef6cf5d50086a9e WatchSource:0}: Error finding container 4e7af37055906f5f9b93e714664dd4940bb74b0893a7da9cdef6cf5d50086a9e: Status 404 returned error can't find the container with id 4e7af37055906f5f9b93e714664dd4940bb74b0893a7da9cdef6cf5d50086a9e
Nov 03 20:15:39.849335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:39.849293    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" event={"ID":"3968f1ea-e378-4dda-bbb2-d83835be0d92","Type":"ContainerStarted","Data":"4e7af37055906f5f9b93e714664dd4940bb74b0893a7da9cdef6cf5d50086a9e"}
Nov 03 20:15:40.856094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.856054    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" event={"ID":"7454d1f9-4e84-42d5-94d8-3b1dd9960841","Type":"ContainerStarted","Data":"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"}
Nov 03 20:15:40.856596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.856132    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:40.857724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.857697    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" event={"ID":"3968f1ea-e378-4dda-bbb2-d83835be0d92","Type":"ContainerStarted","Data":"e4f9043ef57dffb6f1e6677b8a2440fa55425f4fd2f0d45f814c20a80d63e893"}
Nov 03 20:15:40.857849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.857750    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:40.859107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.859086    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" event={"ID":"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8","Type":"ContainerStarted","Data":"8ade6c51abbfd1d15060ce95070a0b595dcfdeb6268f1374c45c1a8f92d91745"}
Nov 03 20:15:40.859221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.859207    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:40.896051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.896005    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" podStartSLOduration=2.033248775 podStartE2EDuration="4.89598492s" totalImagesPullingTime="2.862736145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.894651002 +0000 UTC m=+683.103667001" watchObservedRunningTime="2026-11-03 20:15:40.89598492 +0000 UTC m=+683.105000916"
Nov 03 20:15:40.942582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.936801    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" podStartSLOduration=1.394331646 podStartE2EDuration="2.936783152s" totalImagesPullingTime="1.542451506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.936375003 +0000 UTC m=+683.145391004" watchObservedRunningTime="2026-11-03 20:15:40.936783152 +0000 UTC m=+683.145799149"
Nov 03 20:15:40.974473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:40.974424    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" podStartSLOduration=2.673015936 podStartE2EDuration="2.974411335s" totalImagesPullingTime="301.395399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.973458962 +0000 UTC m=+683.182474960" watchObservedRunningTime="2026-11-03 20:15:40.974411335 +0000 UTC m=+683.183427370"
Nov 03 20:15:42.288358 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:15:42.288324    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:15:46.890865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.890829    5424 generic.go:356] "Generic (PLEG): container finished" podID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerID="cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97" exitCode=0
Nov 03 20:15:46.891322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.890905    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerDied","Data":"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"}
Nov 03 20:15:46.891322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.891284    5424 scope.go:122] "RemoveContainer" containerID="cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"
Nov 03 20:15:46.892602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.892573    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:46.892670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.892642    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:46.892708 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:46.892672    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:47.900214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:47.900183    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerStarted","Data":"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"}
Nov 03 20:15:48.058933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:48.058886    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:48.059130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:48.059109    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:15:48.059176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:48.059161    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:15:49.059174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:49.059138    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:15:49.059581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:49.059190    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:15:50.059482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:50.059439    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:15:50.059904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:50.059502    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:15:51.059414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:51.059372    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:15:51.059623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:51.059433    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:15:51.867979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:51.867948    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:15:51.868148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:51.868026    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:15:51.868235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:51.868213    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:15:52.059252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:52.059223    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:15:52.059427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:52.059276    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:15:53.366430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:53.366397    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:53.367079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:53.367060    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:53.368061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:53.368041    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:15:55.645898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.645863    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:15:55.650342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.650319    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.679283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.679258    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6fc748-openshift-default-dockercfg-h5hq6\""
Nov 03 20:15:55.681353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.681332    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:15:55.717629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717581    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.717840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717640    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.717840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717739    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.717840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717796    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.717840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717823    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvg7\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.718060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717864    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.718060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717890    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.718060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717916    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.718060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.717947    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.818784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818748    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.818784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818785    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818806    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818829    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818848    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvg7\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.818974    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819019    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819049    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819078    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819311    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819379    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819624    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.819881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.819819    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.820513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.820488    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.821420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.821398    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.821809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.821791    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.838121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.838088    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.838280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.838262    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvg7\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:55.963935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:55.963850    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:56.282925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.282898    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:15:56.283916 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:56.283890    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2b8450_64ff_4315_8bb4_cf421e902e43.slice/crio-7f8612b145fa57031ed8374d13563df91941617a0e3d9720bab0e5c7356cfe30 WatchSource:0}: Error finding container 7f8612b145fa57031ed8374d13563df91941617a0e3d9720bab0e5c7356cfe30: Status 404 returned error can't find the container with id 7f8612b145fa57031ed8374d13563df91941617a0e3d9720bab0e5c7356cfe30
Nov 03 20:15:56.286801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.286765    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:56.286866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.286845    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:56.286898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.286885    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:56.653229 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.653135    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:15:56.656422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.656395    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.677363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.677332    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:15:56.686208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.686175    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"openshift-service-ca.crt\""
Nov 03 20:15:56.686208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.686197    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"kube-root-ca.crt\""
Nov 03 20:15:56.713237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.713206    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"default-dockercfg-rxqtx\""
Nov 03 20:15:56.724337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.724306    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"simulator-w2-9973d9-kserve-self-signed-certs\""
Nov 03 20:15:56.729026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.728932    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.729298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.729278    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.729469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.729449    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.729701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.729680    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5728t\" (UniqueName: \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.729959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.729943    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.730095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.730082    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.837186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.837142    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.837444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.837424    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.837720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.837691    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.837876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.837862    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.837993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.837980    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.838106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.838089    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5728t\" (UniqueName: \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.839094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.839060    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.840246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.840230    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.840691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.840673    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.845573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.845507    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.849561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.847438    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.859456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.859424    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5728t\" (UniqueName: \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\") pod \"simulator-w2-9973d9-kserve-86769df68-kjbbt\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") " pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:56.964634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.964542    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" event={"ID":"4e2b8450-64ff-4315-8bb4-cf421e902e43","Type":"ContainerStarted","Data":"b9ca39c2247a2acf78a856135a7fbccdbf04c38cc12159c6d147808ef51bdcd3"}
Nov 03 20:15:56.964634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.964595    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" event={"ID":"4e2b8450-64ff-4315-8bb4-cf421e902e43","Type":"ContainerStarted","Data":"7f8612b145fa57031ed8374d13563df91941617a0e3d9720bab0e5c7356cfe30"}
Nov 03 20:15:56.970963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:56.970940    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:15:57.006680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.006618    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podStartSLOduration=2.006599854 podStartE2EDuration="2.006599854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:57.006205929 +0000 UTC m=+699.215221926" watchObservedRunningTime="2026-11-03 20:15:57.006599854 +0000 UTC m=+699.215615852"
Nov 03 20:15:57.286715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.286680    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:15:57.287003 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:57.286979    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7d0a6a1_5392_493e_b770_d9a6252975db.slice/crio-0c7b5bda9271de5ec3fe5014fcdcf730ce7751ad04734f71f9733f6273e0328c WatchSource:0}: Error finding container 0c7b5bda9271de5ec3fe5014fcdcf730ce7751ad04734f71f9733f6273e0328c: Status 404 returned error can't find the container with id 0c7b5bda9271de5ec3fe5014fcdcf730ce7751ad04734f71f9733f6273e0328c
Nov 03 20:15:57.965478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.965443    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:15:57.966711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.966684    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:15:57.966825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.966736    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:15:57.984987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:57.984950    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" event={"ID":"d7d0a6a1-5392-493e-b770-d9a6252975db","Type":"ContainerStarted","Data":"0c7b5bda9271de5ec3fe5014fcdcf730ce7751ad04734f71f9733f6273e0328c"}
Nov 03 20:15:58.379731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.379700    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:15:58.383401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.383377    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.398436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.398404    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"openshift-service-ca.crt\""
Nov 03 20:15:58.401000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.400971    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:15:58.411196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.411085    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"default-dockercfg-zvkt9\""
Nov 03 20:15:58.411196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.411168    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"kube-root-ca.crt\""
Nov 03 20:15:58.411554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.411491    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"simulator-w4-e80203-kserve-self-signed-certs\""
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.456791    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.456852    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.456935    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.456997    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.457079    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftzm\" (UniqueName: \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.458606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.457175    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558667    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558739    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftzm\" (UniqueName: \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558760    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558785    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558813    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.558863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.558850    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.559207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.559185    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.559390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.559374    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.560179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.560157    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.564421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.564394    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.565630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.565606    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.578941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.578914    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftzm\" (UniqueName: \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\") pod \"simulator-w4-e80203-kserve-5bd485cb8f-8lkhk\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") " pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.700616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.700511    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:15:58.965245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.965169    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:15:58.965245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:58.965222    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:15:59.043857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.043671    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:15:59.047125 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:15:59.047092    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf2c2ad6_ab7b_471d_9311_7b1197f23afb.slice/crio-40cc6fc3cf5d1f9c5c64d5ba7709fdc0ba28ff7c884288a60e8d98544c8733f2 WatchSource:0}: Error finding container 40cc6fc3cf5d1f9c5c64d5ba7709fdc0ba28ff7c884288a60e8d98544c8733f2: Status 404 returned error can't find the container with id 40cc6fc3cf5d1f9c5c64d5ba7709fdc0ba28ff7c884288a60e8d98544c8733f2
Nov 03 20:15:59.965149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.965119    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:15:59.965364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.965173    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:15:59.997393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.997360    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" event={"ID":"cf2c2ad6-ab7b-471d-9311-7b1197f23afb","Type":"ContainerStarted","Data":"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"}
Nov 03 20:15:59.997572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.997411    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" event={"ID":"cf2c2ad6-ab7b-471d-9311-7b1197f23afb","Type":"ContainerStarted","Data":"40cc6fc3cf5d1f9c5c64d5ba7709fdc0ba28ff7c884288a60e8d98544c8733f2"}
Nov 03 20:15:59.998994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:15:59.998968    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" event={"ID":"d7d0a6a1-5392-493e-b770-d9a6252975db","Type":"ContainerStarted","Data":"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"}
Nov 03 20:16:00.059587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.059518    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podStartSLOduration=2.059502752 podStartE2EDuration="2.059502752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.05765741 +0000 UTC m=+702.266673385" watchObservedRunningTime="2026-11-03 20:16:00.059502752 +0000 UTC m=+702.268518789"
Nov 03 20:16:00.103321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.103272    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podStartSLOduration=2.385852447 podStartE2EDuration="4.103259511s" totalImagesPullingTime="1.717407064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.102455642 +0000 UTC m=+702.311471663" watchObservedRunningTime="2026-11-03 20:16:00.103259511 +0000 UTC m=+702.312275504"
Nov 03 20:16:00.597423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.597392    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:16:00.600270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.600245    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.655575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.654242    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"kube-root-ca.crt\""
Nov 03 20:16:00.655575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.654422    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"simulator-w3-482722-kserve-self-signed-certs\""
Nov 03 20:16:00.655575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.654497    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"openshift-service-ca.crt\""
Nov 03 20:16:00.655575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.654569    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"default-dockercfg-hcpz2\""
Nov 03 20:16:00.659003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.658953    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:16:00.683519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683486    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.683710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683571    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm7k\" (UniqueName: \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.683710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683628    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.683841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683774    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.683841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683813    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.683943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.683889    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.703671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.703638    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:16:00.784942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.784864    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.784992    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785011    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785029    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785057    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785087    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm7k\" (UniqueName: \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785506    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.785849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.785823    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.786225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.786108    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.787621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.787591    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.788753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.788247    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.833556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.831391    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm7k\" (UniqueName: \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\") pod \"simulator-w3-482722-kserve-57cdd8f99f-ldrns\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") " pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.914291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.914202    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:00.967811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.964943    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:00.967811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.965001    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:16:00.974327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:00.973209    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:16:01.281712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:01.281686    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:16:01.285475 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:01.285434    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c76d46c_e0b6_466a_9de2_28414bac6a54.slice/crio-18af113f28af5d6094ad1e0f64c2e53ff805205cb6f7c604bb34ddfdb7d8e8e2 WatchSource:0}: Error finding container 18af113f28af5d6094ad1e0f64c2e53ff805205cb6f7c604bb34ddfdb7d8e8e2: Status 404 returned error can't find the container with id 18af113f28af5d6094ad1e0f64c2e53ff805205cb6f7c604bb34ddfdb7d8e8e2
Nov 03 20:16:01.965240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:01.965196    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.965453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:01.965255    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 20:16:02.013910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.013863    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" event={"ID":"6c76d46c-e0b6-466a-9de2-28414bac6a54","Type":"ContainerStarted","Data":"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"}
Nov 03 20:16:02.013910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.013914    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" event={"ID":"6c76d46c-e0b6-466a-9de2-28414bac6a54","Type":"ContainerStarted","Data":"18af113f28af5d6094ad1e0f64c2e53ff805205cb6f7c604bb34ddfdb7d8e8e2"}
Nov 03 20:16:02.056135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.056075    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podStartSLOduration=2.056060275 podStartE2EDuration="2.056060275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:02.054687143 +0000 UTC m=+704.263703153" watchObservedRunningTime="2026-11-03 20:16:02.056060275 +0000 UTC m=+704.265076272"
Nov 03 20:16:02.914938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.914906    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:02.968641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.968614    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:16:02.968958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.968940    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:16:02.969752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:02.969729    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:16:06.971712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:06.971675    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:16:06.985222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:06.985194    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:16:07.048135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:07.048108    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:16:08.700915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:08.700874    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:16:08.713317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:08.713291    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:16:09.057804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:09.057774    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:16:10.385580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.385543    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:16:10.388066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.388050    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.410315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.410281    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9e1bcb-openshift-default-dockercfg-sjgmc\""
Nov 03 20:16:10.425955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.425923    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:16:10.483935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.483904    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.483943    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.483970    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484086    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484120    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484143    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27945\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484159    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484289    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.484358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.484312    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.585786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585740    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585839    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585876    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585923    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585954    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.585979    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586040    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586088    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586124    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27945\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586208    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586310    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586488    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.586598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586583    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.587006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.586982    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.588446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.588417    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.588572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.588495    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.605832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.605804    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.606010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.605992    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27945\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.700184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.700092    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:10.914920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.914877    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:10.928364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:10.928337    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:11.015119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.015093    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:16:11.016228 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:11.016204    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29b35fb_7a74_4fe2_9da4_96b0e5a94966.slice/crio-22a4af44746697a191308852c85d4f6e4d86ce66b02b252255bba688442def07 WatchSource:0}: Error finding container 22a4af44746697a191308852c85d4f6e4d86ce66b02b252255bba688442def07: Status 404 returned error can't find the container with id 22a4af44746697a191308852c85d4f6e4d86ce66b02b252255bba688442def07
Nov 03 20:16:11.018673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.018642    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:11.018747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.018708    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:11.018747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.018734    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:11.059439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.059406    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" event={"ID":"e29b35fb-7a74-4fe2-9da4-96b0e5a94966","Type":"ContainerStarted","Data":"22a4af44746697a191308852c85d4f6e4d86ce66b02b252255bba688442def07"}
Nov 03 20:16:11.071234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:11.071207    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:16:12.067390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:12.067355    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" event={"ID":"e29b35fb-7a74-4fe2-9da4-96b0e5a94966","Type":"ContainerStarted","Data":"fc99d91b67dfada843d8b70e38a1d9339c939c600d51997e9807f31dd2a210e5"}
Nov 03 20:16:12.124556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:12.123708    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podStartSLOduration=2.12369026 podStartE2EDuration="2.12369026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:12.122047341 +0000 UTC m=+714.331063375" watchObservedRunningTime="2026-11-03 20:16:12.12369026 +0000 UTC m=+714.332706257"
Nov 03 20:16:12.289849 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:16:12.289814    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:16:12.700540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:12.700489    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:12.702104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:12.702082    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:12.702201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:12.702140    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 20:16:13.700833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:13.700790    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:13.701346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:13.700864    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 20:16:14.701105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:14.701064    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:14.701620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:14.701122    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 20:16:15.094683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.094644    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:16:15.097940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.097916    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.116175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.116147    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:16:15.145938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.145901    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"openshift-service-ca.crt\""
Nov 03 20:16:15.146428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.146406    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"kube-root-ca.crt\""
Nov 03 20:16:15.146589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.146572    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"default-dockercfg-fr8vq\""
Nov 03 20:16:15.146751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.146636    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"simulator-w5-fd01ad-kserve-self-signed-certs\""
Nov 03 20:16:15.147478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147452    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.147611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147497    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.147665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147645    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.147723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147699    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p62s\" (UniqueName: \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.147929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147903    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.147971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.147945    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249247    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249286    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249319    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249355    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p62s\" (UniqueName: \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249429    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249452    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249778    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249867    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.249900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.249888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.252010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.251977    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.252457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.252440    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.269132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.269106    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p62s\" (UniqueName: \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\") pod \"simulator-w5-fd01ad-kserve-7cfc74bd-zkml4\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") " pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.410607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.410501    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:15.701282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.701208    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:15.701282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.701260    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 20:16:15.744589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:15.742824    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:16:15.748178 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:15.748154    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf42f85c3_00f2_4335_9556_dc8882d2f4fd.slice/crio-adcbd849fde9483d1bc2f666eed46835e6b390086c63836d9da36acdafdcba67 WatchSource:0}: Error finding container adcbd849fde9483d1bc2f666eed46835e6b390086c63836d9da36acdafdcba67: Status 404 returned error can't find the container with id adcbd849fde9483d1bc2f666eed46835e6b390086c63836d9da36acdafdcba67
Nov 03 20:16:16.088637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.088597    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" event={"ID":"f42f85c3-00f2-4335-9556-dc8882d2f4fd","Type":"ContainerStarted","Data":"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"}
Nov 03 20:16:16.088849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.088650    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" event={"ID":"f42f85c3-00f2-4335-9556-dc8882d2f4fd","Type":"ContainerStarted","Data":"adcbd849fde9483d1bc2f666eed46835e6b390086c63836d9da36acdafdcba67"}
Nov 03 20:16:16.134556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.134458    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podStartSLOduration=1.134438239 podStartE2EDuration="1.134438239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:16.132789275 +0000 UTC m=+718.341805272" watchObservedRunningTime="2026-11-03 20:16:16.134438239 +0000 UTC m=+718.343454239"
Nov 03 20:16:16.411655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.411557    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:16.701546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.701441    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:16.701546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:16.701495    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 20:16:17.704984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:17.704953    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:17.705398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:17.705208    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:17.706047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:17.706029    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:16:19.478584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.478550    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:16:19.481759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.481735    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.500031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.499997    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:16:19.509694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.509645    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"openshift-service-ca.crt\""
Nov 03 20:16:19.509827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.509701    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"default-dockercfg-2v8qx\""
Nov 03 20:16:19.510450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.510434    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"simulator-w1-6fc748-kserve-self-signed-certs\""
Nov 03 20:16:19.510505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.510463    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"kube-root-ca.crt\""
Nov 03 20:16:19.598445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598400    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5stm\" (UniqueName: \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.598670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598467    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.598670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598603    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.598808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598670    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.598808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598696    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.598808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.598722    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699782    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5stm\" (UniqueName: \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699859    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699885    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699921    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699942    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.699999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.699975    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.700320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.700299    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.700393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.700362    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.700449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.700402    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.702272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.702252    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.702644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.702623    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.722227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.722189    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5stm\" (UniqueName: \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\") pod \"simulator-w1-6fc748-kserve-b868b885d-82cvh\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") " pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:19.795261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:19.795226    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:20.093097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:20.093064    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:16:20.093282 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:20.093260    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod759a2b86_568a_4b90_8485_2fb28e12a7f2.slice/crio-acc71daaef3850a25aa2479381646516a6033620eba635fe321d898594881a95 WatchSource:0}: Error finding container acc71daaef3850a25aa2479381646516a6033620eba635fe321d898594881a95: Status 404 returned error can't find the container with id acc71daaef3850a25aa2479381646516a6033620eba635fe321d898594881a95
Nov 03 20:16:20.111279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:20.111253    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" event={"ID":"759a2b86-568a-4b90-8485-2fb28e12a7f2","Type":"ContainerStarted","Data":"acc71daaef3850a25aa2479381646516a6033620eba635fe321d898594881a95"}
Nov 03 20:16:21.117466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:21.117432    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" event={"ID":"759a2b86-568a-4b90-8485-2fb28e12a7f2","Type":"ContainerStarted","Data":"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"}
Nov 03 20:16:21.158097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:21.158046    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podStartSLOduration=2.15803443 podStartE2EDuration="2.15803443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:21.157630022 +0000 UTC m=+723.366646041" watchObservedRunningTime="2026-11-03 20:16:21.15803443 +0000 UTC m=+723.367050425"
Nov 03 20:16:21.795629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:21.795579    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:25.411184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.411147    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:25.424262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.424229    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:25.745721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.745686    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:16:25.748867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.748838    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.780682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.780651    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:16:25.783447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.783422    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.783721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.783699    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:16:25.799657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.799626    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:16:25.865869 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.865825    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.865953    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.865984    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866187    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rq2p\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866216    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866261    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866554    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866606    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.866719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.866660    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.967748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967702    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2tq\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.967748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967741    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rq2p\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967778    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967844    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967886    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967947    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.967976    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968000    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968133    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968198    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968229    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968272    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968310    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:25.968617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968340    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968368    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968395    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968658    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968728    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968763    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.968950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.968905    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.969184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.969161    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.969254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.969186    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.969443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.969420    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.971197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.971165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.971378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.971354    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.991094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.991058    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:25.991229 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:25.991165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rq2p\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:26.067593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.067482    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070014    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2tq\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070080    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070137    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070163    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070188    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070223    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070286    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070314    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.070577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070372    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.071050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.070770    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.071050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.071005    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.071431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.071409    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.071674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.071655    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.072226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.072202    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.074437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.074407    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.075618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.075596    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.091493 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.091453    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2tq\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.092945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.092922    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.097943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.097921    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:26.166977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.165979    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:16:26.384109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.384085    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:16:26.384767 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:26.384744    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72029332_2bf5_454c_8c57_ea06343d7908.slice/crio-f207336e62bf93ae8841a88745bbd42f20e61b26e69542da15df39111991757a WatchSource:0}: Error finding container f207336e62bf93ae8841a88745bbd42f20e61b26e69542da15df39111991757a: Status 404 returned error can't find the container with id f207336e62bf93ae8841a88745bbd42f20e61b26e69542da15df39111991757a
Nov 03 20:16:26.387038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.387006    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:26.387123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.387068    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:26.387123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.387097    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:26.416890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.416864    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:16:26.418686 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:26.418659    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8c72c4_0ee1_4b8e_99d7_caf86d381648.slice/crio-fa8a98e3780a02f7e46d08063f5dff15bed839108aff13014c49c7e379c787f1 WatchSource:0}: Error finding container fa8a98e3780a02f7e46d08063f5dff15bed839108aff13014c49c7e379c787f1: Status 404 returned error can't find the container with id fa8a98e3780a02f7e46d08063f5dff15bed839108aff13014c49c7e379c787f1
Nov 03 20:16:26.421197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.421163    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:26.421286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.421245    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:26.421360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:26.421287    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:27.173854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.173754    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" event={"ID":"72029332-2bf5-454c-8c57-ea06343d7908","Type":"ContainerStarted","Data":"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"}
Nov 03 20:16:27.174156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.173861    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" event={"ID":"72029332-2bf5-454c-8c57-ea06343d7908","Type":"ContainerStarted","Data":"f207336e62bf93ae8841a88745bbd42f20e61b26e69542da15df39111991757a"}
Nov 03 20:16:27.175842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.175809    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" event={"ID":"eb8c72c4-0ee1-4b8e-99d7-caf86d381648","Type":"ContainerStarted","Data":"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"}
Nov 03 20:16:27.175976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.175865    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" event={"ID":"eb8c72c4-0ee1-4b8e-99d7-caf86d381648","Type":"ContainerStarted","Data":"fa8a98e3780a02f7e46d08063f5dff15bed839108aff13014c49c7e379c787f1"}
Nov 03 20:16:27.215840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.215787    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podStartSLOduration=2.215774099 podStartE2EDuration="2.215774099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:27.213704044 +0000 UTC m=+729.422720040" watchObservedRunningTime="2026-11-03 20:16:27.215774099 +0000 UTC m=+729.424790096"
Nov 03 20:16:27.261413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:27.261346    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podStartSLOduration=2.261324236 podStartE2EDuration="2.261324236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:27.259036441 +0000 UTC m=+729.468052461" watchObservedRunningTime="2026-11-03 20:16:27.261324236 +0000 UTC m=+729.470340234"
Nov 03 20:16:28.071033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.070859    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:28.072748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.072590    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 20:16:28.072748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.072670    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 20:16:28.098765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.098737    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:28.100123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.100101    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 20:16:28.100230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:28.100143    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 20:16:29.068553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.067891    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 20:16:29.068553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.067968    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 20:16:29.098278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.098237    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 20:16:29.098699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.098297    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 20:16:29.795606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.795564    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:29.808657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:29.808632    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:30.094639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:30.077665    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 20:16:30.094639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:30.077742    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 20:16:30.098618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:30.098585    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 20:16:30.098987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:30.098651    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 20:16:30.207184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:30.207158    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:16:31.068575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:31.067946    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 20:16:31.068575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:31.068013    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 20:16:31.099053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:31.099019    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 20:16:31.099466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:31.099074    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 20:16:32.070555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:32.069902    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 20:16:32.070555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:32.069962    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 20:16:32.098830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:32.098793    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 20:16:32.098995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:32.098849    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 20:16:33.072430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.072397    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:33.072940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.072786    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:33.073450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.073434    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:16:33.103052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.103030    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:33.103326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.103299    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:33.104065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:33.104051    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:16:39.814686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.814642    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:16:39.827429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.827400    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.845272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.845238    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:16:39.845911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.845882    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.846048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.845922    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.846048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.846036    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.846175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.846070    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.846175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.846104    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.846175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.846151    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79j6s\" (UniqueName: \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.854369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.854332    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"kube-root-ca.crt\""
Nov 03 20:16:39.854369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.854366    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"openshift-service-ca.crt\""
Nov 03 20:16:39.854573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.854366    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"default-dockercfg-9ptvd\""
Nov 03 20:16:39.854573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.854432    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"simulator-w6-9e1bcb-kserve-self-signed-certs\""
Nov 03 20:16:39.947034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.946998    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947070    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947110    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79j6s\" (UniqueName: \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947179    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947216    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947306    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947460    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947519    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.947688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.947666    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.949970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.949952    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.950104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.950082    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:39.967291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:39.967268    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79j6s\" (UniqueName: \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\") pod \"simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") " pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:40.152971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:40.152874    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:40.453822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:40.453794    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:16:40.454590 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:40.454563    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb80c215_ac2c_40ef_9759_c22a7c6bd717.slice/crio-c693a9d03ff386e77f944e6c3288a6329fd89ca5941846599a21325fa433d0b4 WatchSource:0}: Error finding container c693a9d03ff386e77f944e6c3288a6329fd89ca5941846599a21325fa433d0b4: Status 404 returned error can't find the container with id c693a9d03ff386e77f944e6c3288a6329fd89ca5941846599a21325fa433d0b4
Nov 03 20:16:41.260065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:41.260023    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" event={"ID":"eb80c215-ac2c-40ef-9759-c22a7c6bd717","Type":"ContainerStarted","Data":"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"}
Nov 03 20:16:41.260065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:41.260081    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" event={"ID":"eb80c215-ac2c-40ef-9759-c22a7c6bd717","Type":"ContainerStarted","Data":"c693a9d03ff386e77f944e6c3288a6329fd89ca5941846599a21325fa433d0b4"}
Nov 03 20:16:41.301755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:41.301698    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podStartSLOduration=2.301679563 podStartE2EDuration="2.301679563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:41.300722748 +0000 UTC m=+743.509738746" watchObservedRunningTime="2026-11-03 20:16:41.301679563 +0000 UTC m=+743.510695564"
Nov 03 20:16:42.153031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:42.152988    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:42.288146 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:16:42.288111    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:16:47.368760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.368722    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:16:47.373279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.373255    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.390440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.390398    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:16:47.400052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.400026    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"premium-w5-fd01ad-kserve-self-signed-certs\""
Nov 03 20:16:47.419078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419047    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.419078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419082    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.419310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419236    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.419369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419311    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.419435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419413    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnwk\" (UniqueName: \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.419492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.419466    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.520994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.520960    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521033    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521116    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnwk\" (UniqueName: \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521156    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521220    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521247    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521662    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521713    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.521981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.521958    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.523355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.523334    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.523874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.523857    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.541774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.541706    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnwk\" (UniqueName: \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\") pod \"premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") " pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.686816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.686728    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:47.984158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:47.983106    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:16:47.984158 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:47.983634    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod245b12cc_7342_4444_8e86_d85ed0e34f5a.slice/crio-382d92ec374a639ed382d794d1150150536a156eff23108cc2061d2cf18d977a WatchSource:0}: Error finding container 382d92ec374a639ed382d794d1150150536a156eff23108cc2061d2cf18d977a: Status 404 returned error can't find the container with id 382d92ec374a639ed382d794d1150150536a156eff23108cc2061d2cf18d977a
Nov 03 20:16:48.302169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:48.302130    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" event={"ID":"245b12cc-7342-4444-8e86-d85ed0e34f5a","Type":"ContainerStarted","Data":"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"}
Nov 03 20:16:48.302365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:48.302179    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" event={"ID":"245b12cc-7342-4444-8e86-d85ed0e34f5a","Type":"ContainerStarted","Data":"382d92ec374a639ed382d794d1150150536a156eff23108cc2061d2cf18d977a"}
Nov 03 20:16:48.342201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:48.342152    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podStartSLOduration=1.342140035 podStartE2EDuration="1.342140035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:48.340210683 +0000 UTC m=+750.549226680" watchObservedRunningTime="2026-11-03 20:16:48.342140035 +0000 UTC m=+750.551156059"
Nov 03 20:16:48.687442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:48.687357    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:50.153089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.153055    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:50.166822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.166798    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:50.323619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.323592    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:16:50.788219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.788178    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:16:50.792624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.792596    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.809393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.809365    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:16:50.819564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.819518    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"openshift-service-ca.crt\""
Nov 03 20:16:50.819877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.819841    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"kube-root-ca.crt\""
Nov 03 20:16:50.822473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.822455    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"premium-w0-e40816-kserve-self-signed-certs\""
Nov 03 20:16:50.822473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.822468    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"default-dockercfg-2vgst\""
Nov 03 20:16:50.853182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853143    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.853333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853201    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.853379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853320    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xfw\" (UniqueName: \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.853379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853359    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.853521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853493    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.853614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.853568    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.954945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.954905    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.954955    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955125    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955163    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955204    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xfw\" (UniqueName: \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955232    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955386    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955548    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.955895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.955867    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.957507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.957487    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.958049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.958023    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:50.974966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:50.974944    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xfw\" (UniqueName: \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\") pod \"premium-w0-e40816-kserve-786ff44b4-7h4pp\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") " pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:51.106298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:51.106213    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:51.415308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:51.415265    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:16:51.415734 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:51.415600    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c126010_cc33_4372_b223_71335546706b.slice/crio-80f036569b14b2ce6f857399a180fe397c1504ecd919a78c3c7b3b3d58397984 WatchSource:0}: Error finding container 80f036569b14b2ce6f857399a180fe397c1504ecd919a78c3c7b3b3d58397984: Status 404 returned error can't find the container with id 80f036569b14b2ce6f857399a180fe397c1504ecd919a78c3c7b3b3d58397984
Nov 03 20:16:52.326069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:52.326030    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" event={"ID":"1c126010-cc33-4372-b223-71335546706b","Type":"ContainerStarted","Data":"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"}
Nov 03 20:16:52.326069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:52.326073    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" event={"ID":"1c126010-cc33-4372-b223-71335546706b","Type":"ContainerStarted","Data":"80f036569b14b2ce6f857399a180fe397c1504ecd919a78c3c7b3b3d58397984"}
Nov 03 20:16:52.365849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:52.365785    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podStartSLOduration=2.365759167 podStartE2EDuration="2.365759167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:52.363111475 +0000 UTC m=+754.572127472" watchObservedRunningTime="2026-11-03 20:16:52.365759167 +0000 UTC m=+754.574775164"
Nov 03 20:16:53.106769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:53.106733    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:16:55.532949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.532914    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:16:55.536599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.536580    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.582403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.582367    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:16:55.608306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608262    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgbw\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608343    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608373    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608478    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608503    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608597    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608657    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608743    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.608960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.608842    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.636576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.636545    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:16:55.640912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.640889    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.677230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.677197    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:16:55.710452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710420    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgbw\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710452    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710475    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710608    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.710744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710653    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.710744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710720    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.710823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710769    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710807    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htm8p\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.710900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710835    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710853    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710865    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710933    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.710994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710968    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.711065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.710993    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.711165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711143    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.711239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711158    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.711239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711202    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.711239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711208    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.711381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711237    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.711381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711265    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.711381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711293    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.711381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711322    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.711776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.711557    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.713186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.713163    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.713683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.713666    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.729548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.729503    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.729733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.729715    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgbw\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.811987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.811904    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.811987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.811946    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.811987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.811962    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812084    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htm8p\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812150    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812202    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812237    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812275    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812300    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812365    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812628    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812672    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.812740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812698    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.813017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.812999    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.814849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.814824    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.814960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.814928    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.831661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.831624    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.831802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.831741    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htm8p\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:55.849900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.849868    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:55.953982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:55.953950    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:56.167651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.167623    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:16:56.167786 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:56.167666    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02498fcf_c866_4b06_a99f_f0ef7e2f31e3.slice/crio-bdd95304074d66068f85c3ca5f92916d214b8ea0a063d39facf3ccf273a89dec WatchSource:0}: Error finding container bdd95304074d66068f85c3ca5f92916d214b8ea0a063d39facf3ccf273a89dec: Status 404 returned error can't find the container with id bdd95304074d66068f85c3ca5f92916d214b8ea0a063d39facf3ccf273a89dec
Nov 03 20:16:56.170277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.170244    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.170353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.170316    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.170391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.170352    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.188760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.188735    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:16:56.194575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.194557    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.211432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.211403    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e80203-openshift-default-dockercfg-6kpjl\""
Nov 03 20:16:56.216785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.216761    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:16:56.262461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.262435    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:16:56.262461 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:56.262441    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437fef2f_8f3c_4208_8a72_54807de6816e.slice/crio-d6150983f9bf3135a72111b86380d2e9ed5f4242367e422b17dc3a20307e43c8 WatchSource:0}: Error finding container d6150983f9bf3135a72111b86380d2e9ed5f4242367e422b17dc3a20307e43c8: Status 404 returned error can't find the container with id d6150983f9bf3135a72111b86380d2e9ed5f4242367e422b17dc3a20307e43c8
Nov 03 20:16:56.265256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.265227    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.265344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.265299    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.265344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.265337    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.325387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325358    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.325501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325481    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.325687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325663    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.325762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325729    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6rp\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.325834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325816    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.325882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325868    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.326012 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.325988    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.326071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.326023    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.326122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.326096    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.352552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.350894    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" event={"ID":"437fef2f-8f3c-4208-8a72-54807de6816e","Type":"ContainerStarted","Data":"3b945ab47bce72de8f2689b8fbae96005359fb8000fc8431615c0807b767ee6b"}
Nov 03 20:16:56.352552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.350963    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" event={"ID":"437fef2f-8f3c-4208-8a72-54807de6816e","Type":"ContainerStarted","Data":"d6150983f9bf3135a72111b86380d2e9ed5f4242367e422b17dc3a20307e43c8"}
Nov 03 20:16:56.366584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.364421    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" event={"ID":"02498fcf-c866-4b06-a99f-f0ef7e2f31e3","Type":"ContainerStarted","Data":"96152187be0103dd2e84e8ee6597060cf1f5cd7b674ae8227ee8a402d2c80483"}
Nov 03 20:16:56.366584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.364548    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" event={"ID":"02498fcf-c866-4b06-a99f-f0ef7e2f31e3","Type":"ContainerStarted","Data":"bdd95304074d66068f85c3ca5f92916d214b8ea0a063d39facf3ccf273a89dec"}
Nov 03 20:16:56.392578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.392243    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podStartSLOduration=1.392228668 podStartE2EDuration="1.392228668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:56.391356696 +0000 UTC m=+758.600372691" watchObservedRunningTime="2026-11-03 20:16:56.392228668 +0000 UTC m=+758.601244657"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427598    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427649    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6rp\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427704    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427757    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427864    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427888    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427914    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.427953    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.428024    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.428380    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.428439    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.428862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.428602    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.429601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.428877    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.429601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.429163    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.431190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.431165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.431311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.431216    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.435443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.435400    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podStartSLOduration=1.435386807 podStartE2EDuration="1.435386807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:56.433818727 +0000 UTC m=+758.642834748" watchObservedRunningTime="2026-11-03 20:16:56.435386807 +0000 UTC m=+758.644402802"
Nov 03 20:16:56.447276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.447251    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6rp\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.447426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.447301    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.519552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.519381    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:56.848558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.847620    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:16:56.850796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.850765    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:16:56.851909 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:16:56.851872    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603a50c1_8df7_41c6_8e8f_df4717091b34.slice/crio-4800abf507115a81be232a2a24ecfe97c5df19fc0f59f2ef37956690065d3484 WatchSource:0}: Error finding container 4800abf507115a81be232a2a24ecfe97c5df19fc0f59f2ef37956690065d3484: Status 404 returned error can't find the container with id 4800abf507115a81be232a2a24ecfe97c5df19fc0f59f2ef37956690065d3484
Nov 03 20:16:56.854513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.854479    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.854616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.854582    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.854665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.854634    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:56.956780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:56.956749    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:16:57.377564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.377487    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" event={"ID":"603a50c1-8df7-41c6-8e8f-df4717091b34","Type":"ContainerStarted","Data":"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"}
Nov 03 20:16:57.377895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.377592    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" event={"ID":"603a50c1-8df7-41c6-8e8f-df4717091b34","Type":"ContainerStarted","Data":"4800abf507115a81be232a2a24ecfe97c5df19fc0f59f2ef37956690065d3484"}
Nov 03 20:16:57.432043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.431982    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podStartSLOduration=1.431962665 podStartE2EDuration="1.431962665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:57.431583567 +0000 UTC m=+759.640599559" watchObservedRunningTime="2026-11-03 20:16:57.431962665 +0000 UTC m=+759.640978663"
Nov 03 20:16:57.522486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.521017    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:16:57.523233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.522832    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:16:57.523233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.522890    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:16:57.690611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.687664    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:57.729550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.729104    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:57.855561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.855426    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:16:57.855561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.855499    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:16:57.961634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.957634    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:57.961634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:57.957718    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:16:58.408666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.408634    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:16:58.531556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.521977    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:16:58.531556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.522041    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:16:58.852962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.852926    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:16:58.853280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.853251    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:16:58.959552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.959150    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:58.959552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:58.959233    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:16:59.521976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.521941    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:16:59.522246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.522222    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:16:59.852615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.851948    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:16:59.852615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.852014    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:16:59.963556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.962650    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:59.963556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:16:59.962762    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:17:00.526556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.522573    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:17:00.526556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.522646    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:17:00.852611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.851943    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:17:00.852611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.852018    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:17:00.958554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.958098    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:17:00.958554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:00.958187    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:17:01.111101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.111006    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:17:01.213759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.213715    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:17:01.499938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.499894    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:17:01.532146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.532112    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:17:01.532468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.532430    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:17:01.854620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.852137    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:17:01.854620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.852204    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:17:01.964768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.964729    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:17:01.965106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:01.965055    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:17:02.525558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:02.521900    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:17:02.525558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:02.521964    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:17:02.958615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:02.957960    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:17:02.958615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:02.958025    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 20:17:03.159557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.158866    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:17:03.165557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.160251    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:17:03.169554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.168023    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:17:03.519821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.519782    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:17:03.520006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.519848    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:17:03.961112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.961032    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:17:03.961585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.961566    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:17:03.962664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:03.962638    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:17:04.524321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:04.524286    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:17:04.524596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:04.524571    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:17:04.525258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:04.525240    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:17:10.729191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.729159    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:17:10.733130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.733114    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.747939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.747919    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9973d9-openshift-default-dockercfg-br8jj\""
Nov 03 20:17:10.764202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.764174    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:17:10.776139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776107    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776254    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776354    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776399    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776437    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776462    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776508    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776661    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.776836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.776813    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.877854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.877818    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878175    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878230    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878264    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878354    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878396    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.878501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.878424    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.879050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.879015    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.879592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.879568    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.879761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.879746    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.880163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.880098    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.884712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.882234    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.884712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.882960    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.891552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.886012    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.895624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.895593    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.896491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.896469    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.906933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.906770    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:10.907078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:10.906987    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:11.046774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.046739    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:11.363924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.363896    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:17:11.364060 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:11.364018    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004f5ef7_6b47_4b5e_ab76_69d20d132603.slice/crio-6621093b88325c4618c780ed6bba28b2c8051d99585731db70ac63a62b72ba33 WatchSource:0}: Error finding container 6621093b88325c4618c780ed6bba28b2c8051d99585731db70ac63a62b72ba33: Status 404 returned error can't find the container with id 6621093b88325c4618c780ed6bba28b2c8051d99585731db70ac63a62b72ba33
Nov 03 20:17:11.366544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.366500    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:11.366652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.366586    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:11.366652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.366614    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:11.501986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.501945    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" event={"ID":"004f5ef7-6b47-4b5e-ab76-69d20d132603","Type":"ContainerStarted","Data":"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"}
Nov 03 20:17:11.502172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.502004    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" event={"ID":"004f5ef7-6b47-4b5e-ab76-69d20d132603","Type":"ContainerStarted","Data":"6621093b88325c4618c780ed6bba28b2c8051d99585731db70ac63a62b72ba33"}
Nov 03 20:17:11.546038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:11.545971    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podStartSLOduration=1.5459564000000001 podStartE2EDuration="1.5459564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:11.545071743 +0000 UTC m=+773.754087739" watchObservedRunningTime="2026-11-03 20:17:11.5459564 +0000 UTC m=+773.754972397"
Nov 03 20:17:12.051312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:12.049955    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:12.053226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:12.052890    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:12.053226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:12.052949    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:12.321713 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:17:12.317879    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:17:13.048567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:13.047824    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:13.048567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:13.047907    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:14.047790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:14.047754    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:14.048210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:14.047819    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:15.048807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:15.048634    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:15.048807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:15.048702    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:16.047277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:16.047241    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:16.047601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:16.047571    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:17.047191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:17.047151    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:17:17.047710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:17.047231    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 20:17:18.150115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.150063    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:18.150725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.150703    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:18.152957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.152926    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:17:18.223146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.223115    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"]
Nov 03 20:17:18.234912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.234869    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.239214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.239174    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"]
Nov 03 20:17:18.250849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.250819    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:18.251101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.250911    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:18.251101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.251009    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:18.251264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.251052    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:18.298283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.298243    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.298480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.298355    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.298595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.298523    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg27v\" (UniqueName: \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.400112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.400081    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.400301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.400136    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.400301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.400230    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg27v\" (UniqueName: \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.400953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.400929    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.402698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.402677    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.419414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.419378    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg27v\" (UniqueName: \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\") pod \"maas-api-revoke-sub-43367f1b-6dkfv\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.571457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.571430    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:18.872860 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:18.872825    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603dda62_655d_4d64_98fc_7397cbf81410.slice/crio-17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6 WatchSource:0}: Error finding container 17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6: Status 404 returned error can't find the container with id 17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6
Nov 03 20:17:18.873093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:18.873074    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"]
Nov 03 20:17:19.580032 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:19.579995    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv" event={"ID":"603dda62-655d-4d64-98fc-7397cbf81410","Type":"ContainerStarted","Data":"17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6"}
Nov 03 20:17:22.601245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:22.601208    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv" event={"ID":"603dda62-655d-4d64-98fc-7397cbf81410","Type":"ContainerStarted","Data":"6f0528363324c96c7c8567bcd52ed61e39bbbacd6901b1d2cac08ee1c63a5610"}
Nov 03 20:17:22.652618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:22.652567    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv" podStartSLOduration=1.564764515 podStartE2EDuration="4.652550232s" totalImagesPullingTime="3.087785717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:22.649729521 +0000 UTC m=+784.858745518" watchObservedRunningTime="2026-11-03 20:17:22.652550232 +0000 UTC m=+784.861566230"
Nov 03 20:17:23.649428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:23.649385    5424 generic.go:356] "Generic (PLEG): container finished" podID="603dda62-655d-4d64-98fc-7397cbf81410" containerID="6f0528363324c96c7c8567bcd52ed61e39bbbacd6901b1d2cac08ee1c63a5610" exitCode=0
Nov 03 20:17:23.650736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:23.650483    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv" event={"ID":"603dda62-655d-4d64-98fc-7397cbf81410","Type":"ContainerDied","Data":"6f0528363324c96c7c8567bcd52ed61e39bbbacd6901b1d2cac08ee1c63a5610"}
Nov 03 20:17:25.169690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.169627    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.173741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.170568    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.173741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.170605    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.276474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.276447    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:25.429595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.429563    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\" (UniqueName: \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\") pod \"603dda62-655d-4d64-98fc-7397cbf81410\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") "
Nov 03 20:17:25.429775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.429687    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\") pod \"603dda62-655d-4d64-98fc-7397cbf81410\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") "
Nov 03 20:17:25.429775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.429736    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\") pod \"603dda62-655d-4d64-98fc-7397cbf81410\" (UID: \"603dda62-655d-4d64-98fc-7397cbf81410\") "
Nov 03 20:17:25.430078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.430052    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca" pod "603dda62-655d-4d64-98fc-7397cbf81410" (UID: "603dda62-655d-4d64-98fc-7397cbf81410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:25.431973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.431949    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca" pod "603dda62-655d-4d64-98fc-7397cbf81410" (UID: "603dda62-655d-4d64-98fc-7397cbf81410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:25.432062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.431974    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v" pod "603dda62-655d-4d64-98fc-7397cbf81410" (UID: "603dda62-655d-4d64-98fc-7397cbf81410"). InnerVolumeSpecName "kube-api-access-wg27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:25.499623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.499590    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:17:25.500107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.500089    5424 cpu_manager.go:414] "containerMap: removing container" podUID="603dda62-655d-4d64-98fc-7397cbf81410" containerName="revoke-keys"
Nov 03 20:17:25.500199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.500109    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603dda62-655d-4d64-98fc-7397cbf81410" containerName="revoke-keys"
Nov 03 20:17:25.500247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.500216    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="603dda62-655d-4d64-98fc-7397cbf81410" containerName="revoke-keys"
Nov 03 20:17:25.503452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.503429    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.523591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.521130    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fd01ad-openshift-default-dockercfg-lbghm\""
Nov 03 20:17:25.524313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.524288    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:17:25.530989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.530959    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531007    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mq25\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531122    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531188    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531293    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531345    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531455    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531554    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531612    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.531738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531722    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:25.531795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531746    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/603dda62-655d-4d64-98fc-7397cbf81410-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:25.531795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.531762    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg27v\" (UniqueName: \"kubernetes.io/projected/603dda62-655d-4d64-98fc-7397cbf81410-kube-api-access-wg27v\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:25.633259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633212    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633258    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633312    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633405    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633447    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633516    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633606    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633627    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mq25\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633668    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.633915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.633808    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.634037 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.634014    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.634107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.634036    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.634214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.634195    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.634214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.634209    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.636387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.636366    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.636641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.636622    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.652465 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.652437    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.652623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.652606    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mq25\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:25.674962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.674925    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"
Nov 03 20:17:25.675147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.674963    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv" event={"ID":"603dda62-655d-4d64-98fc-7397cbf81410","Type":"ContainerDied","Data":"17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6"}
Nov 03 20:17:25.675147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.675022    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17de3bae2efd9024c3478109b6e06aa8055ac1c1d78d48f0531fe0afd40664c6"
Nov 03 20:17:25.819120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:25.819034    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:26.134897 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:26.134857    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18de12c9_5db1_42b4_9f23_a18932e57d95.slice/crio-3f5138b81be63cd76c1c8a9ef5a3b8e39ad37aa0d481cd012e6c7b0fb63d7444 WatchSource:0}: Error finding container 3f5138b81be63cd76c1c8a9ef5a3b8e39ad37aa0d481cd012e6c7b0fb63d7444: Status 404 returned error can't find the container with id 3f5138b81be63cd76c1c8a9ef5a3b8e39ad37aa0d481cd012e6c7b0fb63d7444
Nov 03 20:17:26.135105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.135082    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:17:26.137303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.137270    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:26.137397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.137335    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:26.137397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.137364    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:26.682910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.682870    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" event={"ID":"18de12c9-5db1-42b4-9f23-a18932e57d95","Type":"ContainerStarted","Data":"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"}
Nov 03 20:17:26.683299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.682938    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" event={"ID":"18de12c9-5db1-42b4-9f23-a18932e57d95","Type":"ContainerStarted","Data":"3f5138b81be63cd76c1c8a9ef5a3b8e39ad37aa0d481cd012e6c7b0fb63d7444"}
Nov 03 20:17:26.763274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.763219    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podStartSLOduration=1.7632001320000001 podStartE2EDuration="1.763200132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:26.749743956 +0000 UTC m=+788.958759979" watchObservedRunningTime="2026-11-03 20:17:26.763200132 +0000 UTC m=+788.972216130"
Nov 03 20:17:26.821680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:26.821634    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:27.827556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:27.826658    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:27.827556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:27.826739    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:17:28.824560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:28.821979    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:28.824560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:28.822046    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:17:29.842550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:29.841972    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:29.842550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:29.842033    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:17:30.004773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.004736    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"]
Nov 03 20:17:30.009442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.009424    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.040555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.040185    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:30.040555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.040262    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:30.053376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.053329    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:30.053670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.053641    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:30.059056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.059018    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"]
Nov 03 20:17:30.086844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.086813    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zhd\" (UniqueName: \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.087033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.086894    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.087033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.086970    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.188021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.187935    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.188021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.187998    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zhd\" (UniqueName: \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.188257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.188071    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.188682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.188661    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.190798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.190781    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.209493 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.209463    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zhd\" (UniqueName: \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\") pod \"maas-api-revoke-sub-e65d0869-tzmpc\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.344424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.344392    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:30.659177 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:30.659141    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df06fb2_feda_4f35_aea7_edc251b7c341.slice/crio-12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1 WatchSource:0}: Error finding container 12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1: Status 404 returned error can't find the container with id 12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1
Nov 03 20:17:30.659177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.659159    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"]
Nov 03 20:17:30.708317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.708288    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc" event={"ID":"2df06fb2-feda-4f35-aea7-edc251b7c341","Type":"ContainerStarted","Data":"12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1"}
Nov 03 20:17:30.836033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.834921    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:30.836033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:30.834981    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:17:31.716091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:31.716038    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc" event={"ID":"2df06fb2-feda-4f35-aea7-edc251b7c341","Type":"ContainerStarted","Data":"16856a3c2182e1d9fddd43fc7b46345cb166d559c5d16037d2493643c2a884c7"}
Nov 03 20:17:31.766507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:31.766445    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc" podStartSLOduration=2.766430426 podStartE2EDuration="2.766430426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:31.764829997 +0000 UTC m=+793.973845997" watchObservedRunningTime="2026-11-03 20:17:31.766430426 +0000 UTC m=+793.975446422"
Nov 03 20:17:31.841556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:31.840977    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:31.841556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:31.841044    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:17:32.724907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:32.724873    5424 generic.go:356] "Generic (PLEG): container finished" podID="2df06fb2-feda-4f35-aea7-edc251b7c341" containerID="16856a3c2182e1d9fddd43fc7b46345cb166d559c5d16037d2493643c2a884c7" exitCode=0
Nov 03 20:17:32.725450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:32.724948    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc" event={"ID":"2df06fb2-feda-4f35-aea7-edc251b7c341","Type":"ContainerDied","Data":"16856a3c2182e1d9fddd43fc7b46345cb166d559c5d16037d2493643c2a884c7"}
Nov 03 20:17:32.866763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:32.866727    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:32.867240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:32.867199    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:32.868214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:32.868193    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:17:33.882469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.882440    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:33.938045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.937997    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\") pod \"2df06fb2-feda-4f35-aea7-edc251b7c341\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") "
Nov 03 20:17:33.938250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.938127    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\") pod \"2df06fb2-feda-4f35-aea7-edc251b7c341\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") "
Nov 03 20:17:33.938328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.938300    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\" (UniqueName: \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\") pod \"2df06fb2-feda-4f35-aea7-edc251b7c341\" (UID: \"2df06fb2-feda-4f35-aea7-edc251b7c341\") "
Nov 03 20:17:33.938594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.938567    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca" pod "2df06fb2-feda-4f35-aea7-edc251b7c341" (UID: "2df06fb2-feda-4f35-aea7-edc251b7c341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:33.938831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.938811    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:33.940715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.940674    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd" pod "2df06fb2-feda-4f35-aea7-edc251b7c341" (UID: "2df06fb2-feda-4f35-aea7-edc251b7c341"). InnerVolumeSpecName "kube-api-access-64zhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:33.940715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:33.940700    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca" pod "2df06fb2-feda-4f35-aea7-edc251b7c341" (UID: "2df06fb2-feda-4f35-aea7-edc251b7c341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:34.040515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:34.040477    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df06fb2-feda-4f35-aea7-edc251b7c341-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:34.040515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:34.040505    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zhd\" (UniqueName: \"kubernetes.io/projected/2df06fb2-feda-4f35-aea7-edc251b7c341-kube-api-access-64zhd\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:34.737770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:34.737741    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"
Nov 03 20:17:34.737945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:34.737787    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc" event={"ID":"2df06fb2-feda-4f35-aea7-edc251b7c341","Type":"ContainerDied","Data":"12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1"}
Nov 03 20:17:34.737945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:34.737836    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d0e38ffd0172ce9b116cdd44e3a400e020f5af745df5abe403568589d5dbf1"
Nov 03 20:17:42.286342 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:17:42.286296    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:17:44.320950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.320920    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"]
Nov 03 20:17:44.321364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.321345    5424 cpu_manager.go:414] "containerMap: removing container" podUID="2df06fb2-feda-4f35-aea7-edc251b7c341" containerName="revoke-keys"
Nov 03 20:17:44.321364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.321356    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df06fb2-feda-4f35-aea7-edc251b7c341" containerName="revoke-keys"
Nov 03 20:17:44.321455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.321446    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df06fb2-feda-4f35-aea7-edc251b7c341" containerName="revoke-keys"
Nov 03 20:17:44.324887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.324864    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.345928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.345900    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:44.354022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.353986    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:44.354022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.353986    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:44.354244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.354222    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:44.358471 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.358446    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"]
Nov 03 20:17:44.464348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.464310    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qtz\" (UniqueName: \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.464585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.464369    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.464670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.464591    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.566050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.566006    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.566252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.566163    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qtz\" (UniqueName: \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.566252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.566226    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.566764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.566737    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.568852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.568832    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.586478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.586416    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qtz\" (UniqueName: \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\") pod \"maas-api-revoke-sub-3141c420-pr9qr\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.638387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.638353    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:44.936268 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:44.936236    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d63535_f692_4a43_aa83_b7254035c360.slice/crio-c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13 WatchSource:0}: Error finding container c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13: Status 404 returned error can't find the container with id c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13
Nov 03 20:17:44.937625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:44.937600    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"]
Nov 03 20:17:45.814883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:45.814849    5424 generic.go:356] "Generic (PLEG): container finished" podID="a6d63535-f692-4a43-aa83-b7254035c360" containerID="2fe7da4f7c829c9e4c4df2191e78918b287d0bd25728c906d09a8e2c83d560ac" exitCode=0
Nov 03 20:17:45.815274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:45.814931    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr" event={"ID":"a6d63535-f692-4a43-aa83-b7254035c360","Type":"ContainerDied","Data":"2fe7da4f7c829c9e4c4df2191e78918b287d0bd25728c906d09a8e2c83d560ac"}
Nov 03 20:17:45.815274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:45.815019    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr" event={"ID":"a6d63535-f692-4a43-aa83-b7254035c360","Type":"ContainerStarted","Data":"c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13"}
Nov 03 20:17:46.953402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:46.953373    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:47.066025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.065982    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"]
Nov 03 20:17:47.066423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.066406    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a6d63535-f692-4a43-aa83-b7254035c360" containerName="revoke-keys"
Nov 03 20:17:47.066423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.066420    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d63535-f692-4a43-aa83-b7254035c360" containerName="revoke-keys"
Nov 03 20:17:47.066636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.066514    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d63535-f692-4a43-aa83-b7254035c360" containerName="revoke-keys"
Nov 03 20:17:47.069722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.069702    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.088750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.088722    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"]
Nov 03 20:17:47.097045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.097020    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\") pod \"a6d63535-f692-4a43-aa83-b7254035c360\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") "
Nov 03 20:17:47.097190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.097065    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\" (UniqueName: \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\") pod \"a6d63535-f692-4a43-aa83-b7254035c360\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") "
Nov 03 20:17:47.097281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.097266    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\") pod \"a6d63535-f692-4a43-aa83-b7254035c360\" (UID: \"a6d63535-f692-4a43-aa83-b7254035c360\") "
Nov 03 20:17:47.097557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.097502    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca" pod "a6d63535-f692-4a43-aa83-b7254035c360" (UID: "a6d63535-f692-4a43-aa83-b7254035c360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:47.097921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.097818    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:47.099372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.099348    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz" pod "a6d63535-f692-4a43-aa83-b7254035c360" (UID: "a6d63535-f692-4a43-aa83-b7254035c360"). InnerVolumeSpecName "kube-api-access-j2qtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:47.099633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.099571    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca" pod "a6d63535-f692-4a43-aa83-b7254035c360" (UID: "a6d63535-f692-4a43-aa83-b7254035c360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:47.199187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.199142    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.199392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.199308    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbs6d\" (UniqueName: \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.199450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.199425    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.199593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.199579    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qtz\" (UniqueName: \"kubernetes.io/projected/a6d63535-f692-4a43-aa83-b7254035c360-kube-api-access-j2qtz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:47.199652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.199596    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d63535-f692-4a43-aa83-b7254035c360-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:47.300755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.300719    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbs6d\" (UniqueName: \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.300927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.300793    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.300927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.300919    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.301745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.301719    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.303383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.303361    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.319969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.319939    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbs6d\" (UniqueName: \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\") pod \"maas-api-revoke-sub-f03fd5fb-p87cv\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.383771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.383687    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:47.688240 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:47.688207    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2948a86_7ec2_4a58_9271_65de8f119b37.slice/crio-b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930 WatchSource:0}: Error finding container b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930: Status 404 returned error can't find the container with id b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930
Nov 03 20:17:47.689134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.689106    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"]
Nov 03 20:17:47.827784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.827747    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv" event={"ID":"a2948a86-7ec2-4a58-9271-65de8f119b37","Type":"ContainerStarted","Data":"b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930"}
Nov 03 20:17:47.829143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.829127    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"
Nov 03 20:17:47.829251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.829164    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr" event={"ID":"a6d63535-f692-4a43-aa83-b7254035c360","Type":"ContainerDied","Data":"c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13"}
Nov 03 20:17:47.829251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:47.829202    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44de4a35c06eae8369b9baf8dc259a980aa787a0ff8efa3b6202ac17158ab13"
Nov 03 20:17:48.839155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:48.839053    5424 generic.go:356] "Generic (PLEG): container finished" podID="a2948a86-7ec2-4a58-9271-65de8f119b37" containerID="17f1a703a877c7d5b2d7f2b5d66bd9a838f56332e2fc832055bef87cc3eb5495" exitCode=0
Nov 03 20:17:48.839648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:48.839152    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv" event={"ID":"a2948a86-7ec2-4a58-9271-65de8f119b37","Type":"ContainerDied","Data":"17f1a703a877c7d5b2d7f2b5d66bd9a838f56332e2fc832055bef87cc3eb5495"}
Nov 03 20:17:50.010677 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.010649    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:50.134661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.134578    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\") pod \"a2948a86-7ec2-4a58-9271-65de8f119b37\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") "
Nov 03 20:17:50.134815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.134695    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\") pod \"a2948a86-7ec2-4a58-9271-65de8f119b37\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") "
Nov 03 20:17:50.134815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.134798    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\" (UniqueName: \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\") pod \"a2948a86-7ec2-4a58-9271-65de8f119b37\" (UID: \"a2948a86-7ec2-4a58-9271-65de8f119b37\") "
Nov 03 20:17:50.134939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.134912    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca" pod "a2948a86-7ec2-4a58-9271-65de8f119b37" (UID: "a2948a86-7ec2-4a58-9271-65de8f119b37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:50.135190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.135165    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:50.136964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.136927    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca" pod "a2948a86-7ec2-4a58-9271-65de8f119b37" (UID: "a2948a86-7ec2-4a58-9271-65de8f119b37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:50.137066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.136998    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d" pod "a2948a86-7ec2-4a58-9271-65de8f119b37" (UID: "a2948a86-7ec2-4a58-9271-65de8f119b37"). InnerVolumeSpecName "kube-api-access-tbs6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:50.236500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.236453    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2948a86-7ec2-4a58-9271-65de8f119b37-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:50.236500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.236492    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbs6d\" (UniqueName: \"kubernetes.io/projected/a2948a86-7ec2-4a58-9271-65de8f119b37-kube-api-access-tbs6d\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:50.350547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.350486    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"]
Nov 03 20:17:50.351051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.351031    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a2948a86-7ec2-4a58-9271-65de8f119b37" containerName="revoke-keys"
Nov 03 20:17:50.351150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.351053    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2948a86-7ec2-4a58-9271-65de8f119b37" containerName="revoke-keys"
Nov 03 20:17:50.351205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.351153    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2948a86-7ec2-4a58-9271-65de8f119b37" containerName="revoke-keys"
Nov 03 20:17:50.354470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.354447    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.379974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.379941    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"]
Nov 03 20:17:50.539233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.539181    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.539407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.539375    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkzn\" (UniqueName: \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.539468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.539426    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.640638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.640580    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkzn\" (UniqueName: \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.640819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.640678    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.640819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.640800    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.641470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.641439    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.643154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.643134    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.662986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.662954    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkzn\" (UniqueName: \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\") pod \"maas-api-revoke-sub-81b74f1e-thzpp\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.669874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.669849    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:50.853246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.853165    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"
Nov 03 20:17:50.853393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.853161    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv" event={"ID":"a2948a86-7ec2-4a58-9271-65de8f119b37","Type":"ContainerDied","Data":"b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930"}
Nov 03 20:17:50.853393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.853287    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e99cb595e07401df58c60313face61874f0d2acca97c98969a89e55d2e5930"
Nov 03 20:17:50.975822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:50.975791    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"]
Nov 03 20:17:50.976454 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:50.976427    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf9f8e1_02d2_4000_8724_b5a3dc9adb41.slice/crio-a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b WatchSource:0}: Error finding container a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b: Status 404 returned error can't find the container with id a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b
Nov 03 20:17:51.949429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:51.949384    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp" event={"ID":"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41","Type":"ContainerStarted","Data":"59c01b0bbdff32c6953c578348348966adada897620630161d9e90dba0cdb680"}
Nov 03 20:17:51.950091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:51.950058    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp" event={"ID":"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41","Type":"ContainerStarted","Data":"a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b"}
Nov 03 20:17:52.972566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:52.972376    5424 generic.go:356] "Generic (PLEG): container finished" podID="3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" containerID="59c01b0bbdff32c6953c578348348966adada897620630161d9e90dba0cdb680" exitCode=0
Nov 03 20:17:52.972566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:52.972446    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp" event={"ID":"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41","Type":"ContainerDied","Data":"59c01b0bbdff32c6953c578348348966adada897620630161d9e90dba0cdb680"}
Nov 03 20:17:53.050723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:53.050668    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:17:53.090164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:53.090111    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.191768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:53.191706    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:17:53.221673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:53.221620    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.374377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:53.374240    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:54.125042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.125015    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:54.224000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.223968    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\") pod \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") "
Nov 03 20:17:54.224204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.224033    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\" (UniqueName: \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\") pod \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") "
Nov 03 20:17:54.224204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.224080    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\") pod \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\" (UID: \"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41\") "
Nov 03 20:17:54.224461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.224438    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca" pod "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" (UID: "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:54.224587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.224568    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:54.226287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.226262    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn" pod "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" (UID: "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41"). InnerVolumeSpecName "kube-api-access-xqkzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:54.226392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.226287    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca" pod "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" (UID: "3cf9f8e1-02d2-4000-8724-b5a3dc9adb41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:54.325521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.325483    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:54.325521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.325523    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkzn\" (UniqueName: \"kubernetes.io/projected/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41-kube-api-access-xqkzn\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:54.989589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.989504    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"
Nov 03 20:17:54.989782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.989501    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp" event={"ID":"3cf9f8e1-02d2-4000-8724-b5a3dc9adb41","Type":"ContainerDied","Data":"a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b"}
Nov 03 20:17:54.989782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:54.989645    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0d4f1af1a46bb3dc9d69317263532bed1a48c039e1583c3ebc192c7adb9096b"
Nov 03 20:17:56.071298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.071253    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"]
Nov 03 20:17:56.071801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.071787    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" containerName="revoke-keys"
Nov 03 20:17:56.071801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.071802    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" containerName="revoke-keys"
Nov 03 20:17:56.071883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.071874    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" containerName="revoke-keys"
Nov 03 20:17:56.076298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.076277    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.092276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.092249    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"]
Nov 03 20:17:56.100910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.100883    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:56.100910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.100890    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:56.101063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.100889    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:56.103503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.103467    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:56.148120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.148088    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhn7\" (UniqueName: \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.148296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.148233    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.148364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.148315    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.249314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.249281    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.249477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.249343    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.249477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.249404    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhn7\" (UniqueName: \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.250164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.250098    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.251888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.251869    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.268619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.268599    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhn7\" (UniqueName: \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\") pod \"maas-api-revoke-sub-89e30d0a-wwmw6\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.388917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.388838    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:56.685841 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:56.685807    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6f0f5d_92c0_49ac_bd71_22251da42cea.slice/crio-f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663 WatchSource:0}: Error finding container f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663: Status 404 returned error can't find the container with id f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663
Nov 03 20:17:56.686737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:56.686713    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"]
Nov 03 20:17:57.006327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:57.005370    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6" event={"ID":"3a6f0f5d-92c0-49ac-bd71-22251da42cea","Type":"ContainerStarted","Data":"09a30153287205f33e81632bde9e70fa6e3b4dca96ec911338c82bbc7d4aa586"}
Nov 03 20:17:57.006327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:57.005437    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6" event={"ID":"3a6f0f5d-92c0-49ac-bd71-22251da42cea","Type":"ContainerStarted","Data":"f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663"}
Nov 03 20:17:57.044760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:57.044713    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6" podStartSLOduration=1.044698457 podStartE2EDuration="1.044698457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:57.044510196 +0000 UTC m=+819.253526193" watchObservedRunningTime="2026-11-03 20:17:57.044698457 +0000 UTC m=+819.253714451"
Nov 03 20:17:58.012368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.012330    5424 generic.go:356] "Generic (PLEG): container finished" podID="3a6f0f5d-92c0-49ac-bd71-22251da42cea" containerID="09a30153287205f33e81632bde9e70fa6e3b4dca96ec911338c82bbc7d4aa586" exitCode=0
Nov 03 20:17:58.012822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.012418    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6" event={"ID":"3a6f0f5d-92c0-49ac-bd71-22251da42cea","Type":"ContainerDied","Data":"09a30153287205f33e81632bde9e70fa6e3b4dca96ec911338c82bbc7d4aa586"}
Nov 03 20:17:58.531500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.531469    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"]
Nov 03 20:17:58.534167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.534149    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.555256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.555225    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"]
Nov 03 20:17:58.680830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.678108    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.680830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.678293    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9whx\" (UniqueName: \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.680830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.678453    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.779522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.779481    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.779747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.779574    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9whx\" (UniqueName: \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.779747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.779626    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.780268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.780244    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.782089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.782017    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.798502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.798476    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9whx\" (UniqueName: \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\") pod \"maas-api-revoke-sub-024356e0-8phw8\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:58.845421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:58.845396    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:17:59.144198 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:17:59.144168    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbe545c_d818_482e_ba4e_39347358dfb7.slice/crio-fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5 WatchSource:0}: Error finding container fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5: Status 404 returned error can't find the container with id fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5
Nov 03 20:17:59.144631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.144420    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"]
Nov 03 20:17:59.158156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.158134    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:17:59.286118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.286083    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\") pod \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") "
Nov 03 20:17:59.286279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.286142    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\" (UniqueName: \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\") pod \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") "
Nov 03 20:17:59.286358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.286295    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\") pod \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\" (UID: \"3a6f0f5d-92c0-49ac-bd71-22251da42cea\") "
Nov 03 20:17:59.286693 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.286668    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca" pod "3a6f0f5d-92c0-49ac-bd71-22251da42cea" (UID: "3a6f0f5d-92c0-49ac-bd71-22251da42cea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:59.286893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.286879    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:59.288285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.288269    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca" pod "3a6f0f5d-92c0-49ac-bd71-22251da42cea" (UID: "3a6f0f5d-92c0-49ac-bd71-22251da42cea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:59.288345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.288321    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7" pod "3a6f0f5d-92c0-49ac-bd71-22251da42cea" (UID: "3a6f0f5d-92c0-49ac-bd71-22251da42cea"). InnerVolumeSpecName "kube-api-access-slhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:59.387908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.387883    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6f0f5d-92c0-49ac-bd71-22251da42cea-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:17:59.387908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:17:59.387907    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhn7\" (UniqueName: \"kubernetes.io/projected/3a6f0f5d-92c0-49ac-bd71-22251da42cea-kube-api-access-slhn7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:00.026093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.026058    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8" event={"ID":"0bbe545c-d818-482e-ba4e-39347358dfb7","Type":"ContainerStarted","Data":"32f4478f7e3085ddba07ad267fb1431ef807a6edfe3153ae3052000697174d8e"}
Nov 03 20:18:00.026252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.026124    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8" event={"ID":"0bbe545c-d818-482e-ba4e-39347358dfb7","Type":"ContainerStarted","Data":"fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5"}
Nov 03 20:18:00.027635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.027610    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"
Nov 03 20:18:00.027747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.027631    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6" event={"ID":"3a6f0f5d-92c0-49ac-bd71-22251da42cea","Type":"ContainerDied","Data":"f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663"}
Nov 03 20:18:00.027747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.027691    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f75fb7f88f5b95c6200e2e184e4da3da7a5a6233bbe75a3bb99010f177150663"
Nov 03 20:18:00.065117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.065032    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8" podStartSLOduration=2.065019359 podStartE2EDuration="2.065019359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:00.064454203 +0000 UTC m=+822.273470200" watchObservedRunningTime="2026-11-03 20:18:00.065019359 +0000 UTC m=+822.274035355"
Nov 03 20:18:00.671154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.671123    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"]
Nov 03 20:18:00.671584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.671571    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3a6f0f5d-92c0-49ac-bd71-22251da42cea" containerName="revoke-keys"
Nov 03 20:18:00.671634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.671585    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6f0f5d-92c0-49ac-bd71-22251da42cea" containerName="revoke-keys"
Nov 03 20:18:00.671684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.671676    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6f0f5d-92c0-49ac-bd71-22251da42cea" containerName="revoke-keys"
Nov 03 20:18:00.673689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.673673    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.692631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.692600    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"]
Nov 03 20:18:00.803427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.803397    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbw4\" (UniqueName: \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.803610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.803467    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.803610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.803571    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.905123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.905089    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.905361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.905142    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.905361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.905331    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbw4\" (UniqueName: \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.905877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.905856    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.907719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.907703    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.925372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.925316    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbw4\" (UniqueName: \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\") pod \"maas-api-revoke-sub-7735e22c-bcqvv\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:00.986016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:00.985989    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:01.034688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.034653    5424 generic.go:356] "Generic (PLEG): container finished" podID="0bbe545c-d818-482e-ba4e-39347358dfb7" containerID="32f4478f7e3085ddba07ad267fb1431ef807a6edfe3153ae3052000697174d8e" exitCode=0
Nov 03 20:18:01.034854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.034733    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8" event={"ID":"0bbe545c-d818-482e-ba4e-39347358dfb7","Type":"ContainerDied","Data":"32f4478f7e3085ddba07ad267fb1431ef807a6edfe3153ae3052000697174d8e"}
Nov 03 20:18:01.138935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.138906    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"]
Nov 03 20:18:01.142108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.142086    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.159866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.159844    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"]
Nov 03 20:18:01.284618 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:01.284590    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fec1ace_e27f_4573_9e04_cdf1dbc79ac4.slice/crio-41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d WatchSource:0}: Error finding container 41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d: Status 404 returned error can't find the container with id 41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d
Nov 03 20:18:01.285048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.285024    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"]
Nov 03 20:18:01.309920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.309897    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.310054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.310038    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2kx\" (UniqueName: \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.310104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.310085    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.411148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.411119    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2kx\" (UniqueName: \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.411321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.411176    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.411321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.411211    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.411886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.411862    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.413646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.413628    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.430521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.430497    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2kx\" (UniqueName: \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\") pod \"maas-api-revoke-sub-e47e4d6d-tb4t4\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.455206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.455186    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:01.750827 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:01.750788    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d870b6e_91e5_4348_963a_15bc827e318f.slice/crio-914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42 WatchSource:0}: Error finding container 914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42: Status 404 returned error can't find the container with id 914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42
Nov 03 20:18:01.752138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:01.752115    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"]
Nov 03 20:18:02.042828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.042791    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4" event={"ID":"7d870b6e-91e5-4348-963a-15bc827e318f","Type":"ContainerStarted","Data":"306fcfe19f89a3806ad221a5d8cc2c2ce56fd3c0a6a0b989e20ba367b8290480"}
Nov 03 20:18:02.042980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.042854    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4" event={"ID":"7d870b6e-91e5-4348-963a-15bc827e318f","Type":"ContainerStarted","Data":"914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42"}
Nov 03 20:18:02.044383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.044355    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv" event={"ID":"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4","Type":"ContainerStarted","Data":"115425ba0f028ebf0dbc03474e25fdf1eef555d1e1aae460969cf045d596e16d"}
Nov 03 20:18:02.044495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.044407    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv" event={"ID":"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4","Type":"ContainerStarted","Data":"41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d"}
Nov 03 20:18:02.086456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.086403    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4" podStartSLOduration=1.086389689 podStartE2EDuration="1.086389689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:02.084868773 +0000 UTC m=+824.293884771" watchObservedRunningTime="2026-11-03 20:18:02.086389689 +0000 UTC m=+824.295405680"
Nov 03 20:18:02.125353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.125298    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv" podStartSLOduration=2.125281834 podStartE2EDuration="2.125281834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:02.122273432 +0000 UTC m=+824.331289426" watchObservedRunningTime="2026-11-03 20:18:02.125281834 +0000 UTC m=+824.334297830"
Nov 03 20:18:02.234995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.234969    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:18:02.324310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.324235    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\") pod \"0bbe545c-d818-482e-ba4e-39347358dfb7\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") "
Nov 03 20:18:02.324459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.324345    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\") pod \"0bbe545c-d818-482e-ba4e-39347358dfb7\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") "
Nov 03 20:18:02.324459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.324394    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\" (UniqueName: \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\") pod \"0bbe545c-d818-482e-ba4e-39347358dfb7\" (UID: \"0bbe545c-d818-482e-ba4e-39347358dfb7\") "
Nov 03 20:18:02.324753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.324730    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca" pod "0bbe545c-d818-482e-ba4e-39347358dfb7" (UID: "0bbe545c-d818-482e-ba4e-39347358dfb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:02.324930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.324893    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:02.326411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.326381    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca" pod "0bbe545c-d818-482e-ba4e-39347358dfb7" (UID: "0bbe545c-d818-482e-ba4e-39347358dfb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:02.326582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.326565    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx" pod "0bbe545c-d818-482e-ba4e-39347358dfb7" (UID: "0bbe545c-d818-482e-ba4e-39347358dfb7"). InnerVolumeSpecName "kube-api-access-c9whx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:02.425926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.425900    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bbe545c-d818-482e-ba4e-39347358dfb7-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:02.425926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:02.425926    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9whx\" (UniqueName: \"kubernetes.io/projected/0bbe545c-d818-482e-ba4e-39347358dfb7-kube-api-access-c9whx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:03.052423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.052393    5424 generic.go:356] "Generic (PLEG): container finished" podID="7d870b6e-91e5-4348-963a-15bc827e318f" containerID="306fcfe19f89a3806ad221a5d8cc2c2ce56fd3c0a6a0b989e20ba367b8290480" exitCode=0
Nov 03 20:18:03.052912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.052475    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4" event={"ID":"7d870b6e-91e5-4348-963a-15bc827e318f","Type":"ContainerDied","Data":"306fcfe19f89a3806ad221a5d8cc2c2ce56fd3c0a6a0b989e20ba367b8290480"}
Nov 03 20:18:03.054217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.054190    5424 generic.go:356] "Generic (PLEG): container finished" podID="2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" containerID="115425ba0f028ebf0dbc03474e25fdf1eef555d1e1aae460969cf045d596e16d" exitCode=0
Nov 03 20:18:03.054345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.054271    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv" event={"ID":"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4","Type":"ContainerDied","Data":"115425ba0f028ebf0dbc03474e25fdf1eef555d1e1aae460969cf045d596e16d"}
Nov 03 20:18:03.055609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.055589    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8" event={"ID":"0bbe545c-d818-482e-ba4e-39347358dfb7","Type":"ContainerDied","Data":"fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5"}
Nov 03 20:18:03.055699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.055625    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fce500e0826056bb7f7360c06cf91b4b9a929af05ba07f8572631ee12883f9e5"
Nov 03 20:18:03.055699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:03.055598    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"
Nov 03 20:18:04.221160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.221131    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:04.246971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.246945    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:04.348319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348227    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\" (UniqueName: \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\") pod \"7d870b6e-91e5-4348-963a-15bc827e318f\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") "
Nov 03 20:18:04.348319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348282    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\" (UniqueName: \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\") pod \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") "
Nov 03 20:18:04.348594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348363    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\") pod \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") "
Nov 03 20:18:04.348594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348518    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\") pod \"7d870b6e-91e5-4348-963a-15bc827e318f\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") "
Nov 03 20:18:04.348719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348643    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\") pod \"7d870b6e-91e5-4348-963a-15bc827e318f\" (UID: \"7d870b6e-91e5-4348-963a-15bc827e318f\") "
Nov 03 20:18:04.348719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348706    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\") pod \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\" (UID: \"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4\") "
Nov 03 20:18:04.348952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.348924    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca" pod "7d870b6e-91e5-4348-963a-15bc827e318f" (UID: "7d870b6e-91e5-4348-963a-15bc827e318f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:04.349149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.349036    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca" pod "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" (UID: "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:04.349386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.349363    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.349464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.349393    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.350846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.350823    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx" pod "7d870b6e-91e5-4348-963a-15bc827e318f" (UID: "7d870b6e-91e5-4348-963a-15bc827e318f"). InnerVolumeSpecName "kube-api-access-fv2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:04.350846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.350833    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4" pod "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" (UID: "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4"). InnerVolumeSpecName "kube-api-access-7cbw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:04.351213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.351189    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca" pod "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" (UID: "2fec1ace-e27f-4573-9e04-cdf1dbc79ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:04.351213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.351203    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca" pod "7d870b6e-91e5-4348-963a-15bc827e318f" (UID: "7d870b6e-91e5-4348-963a-15bc827e318f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:04.450117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.450081    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d870b6e-91e5-4348-963a-15bc827e318f-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.450117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.450113    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv2kx\" (UniqueName: \"kubernetes.io/projected/7d870b6e-91e5-4348-963a-15bc827e318f-kube-api-access-fv2kx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.450117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.450123    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbw4\" (UniqueName: \"kubernetes.io/projected/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-kube-api-access-7cbw4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.450347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:04.450133    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:05.073883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.073848    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv" event={"ID":"2fec1ace-e27f-4573-9e04-cdf1dbc79ac4","Type":"ContainerDied","Data":"41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d"}
Nov 03 20:18:05.074057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.073886    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"
Nov 03 20:18:05.074057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.073909    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41d103ad5c92838fb968c23afbde701dd27d3f77dc5ef07bcc7d3f765153181d"
Nov 03 20:18:05.075407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.075377    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4" event={"ID":"7d870b6e-91e5-4348-963a-15bc827e318f","Type":"ContainerDied","Data":"914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42"}
Nov 03 20:18:05.075524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.075412    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"
Nov 03 20:18:05.075524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:05.075426    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914fed733c0a12dd28ce08ddaa828913a020ca1c23d4ee0dad3c2204f992bd42"
Nov 03 20:18:11.042121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042084    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"]
Nov 03 20:18:11.042560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042519    5424 cpu_manager.go:414] "containerMap: removing container" podUID="2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" containerName="revoke-keys"
Nov 03 20:18:11.042560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042548    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" containerName="revoke-keys"
Nov 03 20:18:11.042658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042571    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0bbe545c-d818-482e-ba4e-39347358dfb7" containerName="revoke-keys"
Nov 03 20:18:11.042658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042576    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbe545c-d818-482e-ba4e-39347358dfb7" containerName="revoke-keys"
Nov 03 20:18:11.042658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042583    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7d870b6e-91e5-4348-963a-15bc827e318f" containerName="revoke-keys"
Nov 03 20:18:11.042658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042588    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d870b6e-91e5-4348-963a-15bc827e318f" containerName="revoke-keys"
Nov 03 20:18:11.042658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042656    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" containerName="revoke-keys"
Nov 03 20:18:11.042798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042667    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbe545c-d818-482e-ba4e-39347358dfb7" containerName="revoke-keys"
Nov 03 20:18:11.042798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.042672    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d870b6e-91e5-4348-963a-15bc827e318f" containerName="revoke-keys"
Nov 03 20:18:11.045952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.045932    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.063024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.062994    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"]
Nov 03 20:18:11.071434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.071408    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:11.071621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.071474    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:18:11.071621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.071588    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:18:11.071746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.071641    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:18:11.100083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.100050    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s74c\" (UniqueName: \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.100247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.100112    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.100247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.100209    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.201482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.201431    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.201679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.201516    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.201737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.201676    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s74c\" (UniqueName: \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.202180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.202141    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.204058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.204033    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.220732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.220707    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s74c\" (UniqueName: \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\") pod \"maas-api-revoke-sub-fd4cce80-wpzvp\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.360807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.360731    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:11.666859 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:11.666829    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c8452f_59d8_43b7_a1c6_2c56e398e105.slice/crio-0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f WatchSource:0}: Error finding container 0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f: Status 404 returned error can't find the container with id 0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f
Nov 03 20:18:11.666998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:11.666977    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"]
Nov 03 20:18:12.142763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.142720    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp" event={"ID":"44c8452f-59d8-43b7-a1c6-2c56e398e105","Type":"ContainerStarted","Data":"53d00490ad5b25755a300726af5b1ee2251d7bf6871a3dd0bf2ad6383c2f84f8"}
Nov 03 20:18:12.143189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.142789    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp" event={"ID":"44c8452f-59d8-43b7-a1c6-2c56e398e105","Type":"ContainerStarted","Data":"0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f"}
Nov 03 20:18:12.182141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.182046    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp" podStartSLOduration=1.182031738 podStartE2EDuration="1.182031738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:12.180144091 +0000 UTC m=+834.389160099" watchObservedRunningTime="2026-11-03 20:18:12.182031738 +0000 UTC m=+834.391047767"
Nov 03 20:18:12.285823 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:18:12.285790    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:18:12.758448 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.758414    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"]
Nov 03 20:18:12.762113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.762097    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.781478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.781445    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"]
Nov 03 20:18:12.823733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.823693    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.823878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.823822    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2gr\" (UniqueName: \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.823932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.823881    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.924887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.924852    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.925062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.924934    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2gr\" (UniqueName: \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.925062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.924978    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.925442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.925421    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.927362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.927341    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:12.945471 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:12.945442    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2gr\" (UniqueName: \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\") pod \"maas-api-revoke-sub-7144e1e8-2vmm9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:13.075012 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:13.074922    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:13.152171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:13.152141    5424 generic.go:356] "Generic (PLEG): container finished" podID="44c8452f-59d8-43b7-a1c6-2c56e398e105" containerID="53d00490ad5b25755a300726af5b1ee2251d7bf6871a3dd0bf2ad6383c2f84f8" exitCode=0
Nov 03 20:18:13.152654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:13.152238    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp" event={"ID":"44c8452f-59d8-43b7-a1c6-2c56e398e105","Type":"ContainerDied","Data":"53d00490ad5b25755a300726af5b1ee2251d7bf6871a3dd0bf2ad6383c2f84f8"}
Nov 03 20:18:13.376300 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:13.376267    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74526ef8_b63b_45da_b6b7_c1bdbf28e8b9.slice/crio-24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298 WatchSource:0}: Error finding container 24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298: Status 404 returned error can't find the container with id 24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298
Nov 03 20:18:13.377515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:13.377488    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"]
Nov 03 20:18:14.159748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.159711    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9" event={"ID":"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9","Type":"ContainerStarted","Data":"469dfa46bad7966ebe23064b74bde09e6ff0357d9930e2fcb3d5204a1c0062c4"}
Nov 03 20:18:14.160155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.159768    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9" event={"ID":"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9","Type":"ContainerStarted","Data":"24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298"}
Nov 03 20:18:14.201398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.201329    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9" podStartSLOduration=2.201309893 podStartE2EDuration="2.201309893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:14.199709158 +0000 UTC m=+836.408725170" watchObservedRunningTime="2026-11-03 20:18:14.201309893 +0000 UTC m=+836.410325890"
Nov 03 20:18:14.321593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.321561    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:14.342101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.342065    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\") pod \"44c8452f-59d8-43b7-a1c6-2c56e398e105\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") "
Nov 03 20:18:14.342289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.342188    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\" (UniqueName: \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\") pod \"44c8452f-59d8-43b7-a1c6-2c56e398e105\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") "
Nov 03 20:18:14.342289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.342235    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\") pod \"44c8452f-59d8-43b7-a1c6-2c56e398e105\" (UID: \"44c8452f-59d8-43b7-a1c6-2c56e398e105\") "
Nov 03 20:18:14.342697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.342670    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca" pod "44c8452f-59d8-43b7-a1c6-2c56e398e105" (UID: "44c8452f-59d8-43b7-a1c6-2c56e398e105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:14.344385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.344356    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca" pod "44c8452f-59d8-43b7-a1c6-2c56e398e105" (UID: "44c8452f-59d8-43b7-a1c6-2c56e398e105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:14.344582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.344567    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c" pod "44c8452f-59d8-43b7-a1c6-2c56e398e105" (UID: "44c8452f-59d8-43b7-a1c6-2c56e398e105"). InnerVolumeSpecName "kube-api-access-7s74c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:14.353117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.353081    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"]
Nov 03 20:18:14.353846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.353824    5424 cpu_manager.go:414] "containerMap: removing container" podUID="44c8452f-59d8-43b7-a1c6-2c56e398e105" containerName="revoke-keys"
Nov 03 20:18:14.353899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.353849    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c8452f-59d8-43b7-a1c6-2c56e398e105" containerName="revoke-keys"
Nov 03 20:18:14.353994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.353983    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c8452f-59d8-43b7-a1c6-2c56e398e105" containerName="revoke-keys"
Nov 03 20:18:14.357661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.357632    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.373105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.373072    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"]
Nov 03 20:18:14.443251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443168    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896xz\" (UniqueName: \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.443251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443228    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.443455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443382    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.443501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443478    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:14.443501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443492    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s74c\" (UniqueName: \"kubernetes.io/projected/44c8452f-59d8-43b7-a1c6-2c56e398e105-kube-api-access-7s74c\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:14.443593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.443501    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c8452f-59d8-43b7-a1c6-2c56e398e105-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:14.544898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.544857    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.545081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.544982    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.545081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.545056    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896xz\" (UniqueName: \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.545684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.545654    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.547418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.547396    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.564541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.564509    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896xz\" (UniqueName: \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\") pod \"maas-api-revoke-sub-a740d1b5-mkjs2\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.682962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.682918    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:14.988483 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:14.988446    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337f958e_4d25_452f_97f2_230370c2f6bd.slice/crio-7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055 WatchSource:0}: Error finding container 7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055: Status 404 returned error can't find the container with id 7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055
Nov 03 20:18:14.988868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:14.988848    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"]
Nov 03 20:18:15.000567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.000514    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:18:15.006061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.006031    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.023849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.023809    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"distinct-w6-9e1bcb-kserve-self-signed-certs\""
Nov 03 20:18:15.030210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.030174    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:18:15.051756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.051725    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.051916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.051761    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8v2\" (UniqueName: \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.051979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.051907    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.052057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.051980    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.052147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.052130    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.052208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.052184    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153149    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153282    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153336    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153375    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153408    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8v2\" (UniqueName: \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153520    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153636    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.153929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.153904    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.154599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.154571    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.155995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.155972    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.156486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.156468    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.168747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.168718    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"
Nov 03 20:18:15.168747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.168736    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp" event={"ID":"44c8452f-59d8-43b7-a1c6-2c56e398e105","Type":"ContainerDied","Data":"0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f"}
Nov 03 20:18:15.169270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.168792    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0108a432167e7cef6c4dfb6af50d17a847649cff5943c2805d407fc2ad18df2f"
Nov 03 20:18:15.170155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.170125    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2" event={"ID":"337f958e-4d25-452f-97f2-230370c2f6bd","Type":"ContainerStarted","Data":"7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055"}
Nov 03 20:18:15.171945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.171919    5424 generic.go:356] "Generic (PLEG): container finished" podID="74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" containerID="469dfa46bad7966ebe23064b74bde09e6ff0357d9930e2fcb3d5204a1c0062c4" exitCode=0
Nov 03 20:18:15.172073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.171974    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9" event={"ID":"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9","Type":"ContainerDied","Data":"469dfa46bad7966ebe23064b74bde09e6ff0357d9930e2fcb3d5204a1c0062c4"}
Nov 03 20:18:15.173441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.173415    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8v2\" (UniqueName: \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\") pod \"distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.337939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.337898    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:15.644913 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:15.644875    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4122f8e_788d_4843_be7e_8bbcf95cb094.slice/crio-53a75922cf2705017fcc02ee7c8fccd7ed7896cc6dfeb5c80c5a5aef48867b00 WatchSource:0}: Error finding container 53a75922cf2705017fcc02ee7c8fccd7ed7896cc6dfeb5c80c5a5aef48867b00: Status 404 returned error can't find the container with id 53a75922cf2705017fcc02ee7c8fccd7ed7896cc6dfeb5c80c5a5aef48867b00
Nov 03 20:18:15.645711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:15.645686    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:18:16.180930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.180892    5424 generic.go:356] "Generic (PLEG): container finished" podID="337f958e-4d25-452f-97f2-230370c2f6bd" containerID="faef4cefcf22762fdf6e1b84b40c6fd09a23773b61b762bb8e84f2872495288c" exitCode=0
Nov 03 20:18:16.181402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.180989    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2" event={"ID":"337f958e-4d25-452f-97f2-230370c2f6bd","Type":"ContainerDied","Data":"faef4cefcf22762fdf6e1b84b40c6fd09a23773b61b762bb8e84f2872495288c"}
Nov 03 20:18:16.182911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.182881    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" event={"ID":"d4122f8e-788d-4843-be7e-8bbcf95cb094","Type":"ContainerStarted","Data":"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"}
Nov 03 20:18:16.183051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.182922    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" event={"ID":"d4122f8e-788d-4843-be7e-8bbcf95cb094","Type":"ContainerStarted","Data":"53a75922cf2705017fcc02ee7c8fccd7ed7896cc6dfeb5c80c5a5aef48867b00"}
Nov 03 20:18:16.258342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.258279    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podStartSLOduration=2.258256787 podStartE2EDuration="2.258256787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:16.25692101 +0000 UTC m=+838.465936989" watchObservedRunningTime="2026-11-03 20:18:16.258256787 +0000 UTC m=+838.467272785"
Nov 03 20:18:16.338810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.338781    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:16.340054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.340035    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:16.377919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.377882    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\") pod \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") "
Nov 03 20:18:16.378110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.377945    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\") pod \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") "
Nov 03 20:18:16.378110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.378080    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\" (UniqueName: \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\") pod \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\" (UID: \"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9\") "
Nov 03 20:18:16.378550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.378489    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca" pod "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" (UID: "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:16.378836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.378709    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:16.380688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.380659    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca" pod "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" (UID: "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:16.380826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.380808    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr" pod "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" (UID: "74526ef8-b63b-45da-b6b7-c1bdbf28e8b9"). InnerVolumeSpecName "kube-api-access-6t2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:16.480482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.480393    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:16.480482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:16.480425    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t2gr\" (UniqueName: \"kubernetes.io/projected/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9-kube-api-access-6t2gr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.191147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.191116    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"
Nov 03 20:18:17.191635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.191151    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9" event={"ID":"74526ef8-b63b-45da-b6b7-c1bdbf28e8b9","Type":"ContainerDied","Data":"24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298"}
Nov 03 20:18:17.191635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.191199    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24db2d1103bad9eff82a405c7cf2164aedae2ad910d9f2be5972207bf7036298"
Nov 03 20:18:17.346167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.346139    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:17.392431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.392393    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\") pod \"337f958e-4d25-452f-97f2-230370c2f6bd\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") "
Nov 03 20:18:17.392745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.392458    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\" (UniqueName: \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\") pod \"337f958e-4d25-452f-97f2-230370c2f6bd\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") "
Nov 03 20:18:17.392745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.392652    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\") pod \"337f958e-4d25-452f-97f2-230370c2f6bd\" (UID: \"337f958e-4d25-452f-97f2-230370c2f6bd\") "
Nov 03 20:18:17.393179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.393156    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca" pod "337f958e-4d25-452f-97f2-230370c2f6bd" (UID: "337f958e-4d25-452f-97f2-230370c2f6bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:17.395219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.395192    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz" pod "337f958e-4d25-452f-97f2-230370c2f6bd" (UID: "337f958e-4d25-452f-97f2-230370c2f6bd"). InnerVolumeSpecName "kube-api-access-896xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:17.395324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.395236    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca" pod "337f958e-4d25-452f-97f2-230370c2f6bd" (UID: "337f958e-4d25-452f-97f2-230370c2f6bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:17.494810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.494779    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.494810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.494806    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896xz\" (UniqueName: \"kubernetes.io/projected/337f958e-4d25-452f-97f2-230370c2f6bd-kube-api-access-896xz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.494810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:17.494816    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337f958e-4d25-452f-97f2-230370c2f6bd-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:18.199542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:18.199440    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"
Nov 03 20:18:18.199542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:18.199476    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2" event={"ID":"337f958e-4d25-452f-97f2-230370c2f6bd","Type":"ContainerDied","Data":"7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055"}
Nov 03 20:18:18.199542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:18.199539    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7273500c86beb9292b98541e8f884bd66e620c4974805e76fcbc9f06c3778055"
Nov 03 20:18:22.502495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.502450    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"]
Nov 03 20:18:22.503755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.503727    5424 cpu_manager.go:414] "containerMap: removing container" podUID="337f958e-4d25-452f-97f2-230370c2f6bd" containerName="revoke-keys"
Nov 03 20:18:22.503906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.503894    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337f958e-4d25-452f-97f2-230370c2f6bd" containerName="revoke-keys"
Nov 03 20:18:22.504021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.504012    5424 cpu_manager.go:414] "containerMap: removing container" podUID="74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" containerName="revoke-keys"
Nov 03 20:18:22.504081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.504073    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" containerName="revoke-keys"
Nov 03 20:18:22.504258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.504249    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="337f958e-4d25-452f-97f2-230370c2f6bd" containerName="revoke-keys"
Nov 03 20:18:22.504331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.504324    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" containerName="revoke-keys"
Nov 03 20:18:22.530259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.530223    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"]
Nov 03 20:18:22.530609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.530589    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.568557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.566857    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:22.568557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.566970    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:18:22.568557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.567078    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:18:22.568557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.567181    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:18:22.695558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.670201    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.695558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.670287    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8ml\" (UniqueName: \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.695558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.670350    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.805613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.771215    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8ml\" (UniqueName: \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.805613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.771308    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.805613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.771494    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.822319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.806140    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.869145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.869061    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.885557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.882219    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8ml\" (UniqueName: \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\") pod \"maas-api-revoke-sub-6544f52d-twf5p\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:22.960555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:22.960416    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:23.330804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:23.330751    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.866609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:23.866500    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"]
Nov 03 20:18:24.321556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:24.317892    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p" event={"ID":"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a","Type":"ContainerStarted","Data":"5f9444641f742d506967405cba0e60c84560eef9e74bbe6acf489782fa68c7ff"}
Nov 03 20:18:25.332111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:25.332069    5424 generic.go:356] "Generic (PLEG): container finished" podID="cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" containerID="41aa8b8f40ee4e54e75540f00776a2a542195d55f6548464fce5d42cb781e25b" exitCode=0
Nov 03 20:18:25.332592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:25.332358    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p" event={"ID":"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a","Type":"ContainerDied","Data":"41aa8b8f40ee4e54e75540f00776a2a542195d55f6548464fce5d42cb781e25b"}
Nov 03 20:18:25.348087 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:25.348054    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:25.373470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:25.373438    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:26.350022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.349995    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:18:26.486125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.486095    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:18:26.486632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.486616    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:26.489813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.489786    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.513777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.513740    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:18:26.520034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520006    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\") pod \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") "
Nov 03 20:18:26.520179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520114    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\" (UniqueName: \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\") pod \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") "
Nov 03 20:18:26.520265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520215    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\") pod \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\" (UID: \"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a\") "
Nov 03 20:18:26.520393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520373    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520438    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ktk\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520591    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520599    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca" pod "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" (UID: "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:26.520813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520792    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520839    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520877    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.520972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520956    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.521033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.520983    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.521033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.521009    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.521127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.521099    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:26.522767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.522740    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml" pod "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" (UID: "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a"). InnerVolumeSpecName "kube-api-access-kr8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:26.526131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.526015    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca" pod "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" (UID: "cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:26.622488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622393    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622437    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622457    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622501    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622517    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622566    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622619    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622668    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ktk\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.622804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622714    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.623079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622831    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:26.623079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622849    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8ml\" (UniqueName: \"kubernetes.io/projected/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a-kube-api-access-kr8ml\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:18:26.623079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.622928    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.623079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.623012    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.623279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.623138    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.623279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.623177    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.623574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.623276    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.625054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.625029    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.625194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.625168    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.641373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.641342    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.641564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.641519    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ktk\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:26.802902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:26.802860    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:27.122693 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:27.122662    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d86104_3352_47ae_8f18_1e0b7effee00.slice/crio-e256af7d39cefefe5d6ef0313999ac4fea2acfeb5579978b72d1db07ed369475 WatchSource:0}: Error finding container e256af7d39cefefe5d6ef0313999ac4fea2acfeb5579978b72d1db07ed369475: Status 404 returned error can't find the container with id e256af7d39cefefe5d6ef0313999ac4fea2acfeb5579978b72d1db07ed369475
Nov 03 20:18:27.124254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.124226    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:18:27.125407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.125375    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:27.125457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.125441    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:27.125487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.125472    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:27.346562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.346508    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" event={"ID":"86d86104-3352-47ae-8f18-1e0b7effee00","Type":"ContainerStarted","Data":"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"}
Nov 03 20:18:27.346743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.346588    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" event={"ID":"86d86104-3352-47ae-8f18-1e0b7effee00","Type":"ContainerStarted","Data":"e256af7d39cefefe5d6ef0313999ac4fea2acfeb5579978b72d1db07ed369475"}
Nov 03 20:18:27.347922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.347901    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"
Nov 03 20:18:27.348051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.347922    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p" event={"ID":"cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a","Type":"ContainerDied","Data":"5f9444641f742d506967405cba0e60c84560eef9e74bbe6acf489782fa68c7ff"}
Nov 03 20:18:27.348051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.347976    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f9444641f742d506967405cba0e60c84560eef9e74bbe6acf489782fa68c7ff"
Nov 03 20:18:27.388645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.388519    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podStartSLOduration=1.388503118 podStartE2EDuration="1.388503118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:27.388064489 +0000 UTC m=+849.597080486" watchObservedRunningTime="2026-11-03 20:18:27.388503118 +0000 UTC m=+849.597519116"
Nov 03 20:18:27.783812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.783785    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"]
Nov 03 20:18:27.784756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.784730    5424 cpu_manager.go:414] "containerMap: removing container" podUID="cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" containerName="revoke-keys"
Nov 03 20:18:27.784934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.784919    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" containerName="revoke-keys"
Nov 03 20:18:27.785233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.785214    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" containerName="revoke-keys"
Nov 03 20:18:27.795942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.795905    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.817343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.805695    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:27.817343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.810796    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"]
Nov 03 20:18:27.852613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.852520    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.852779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.852622    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.852838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.852785    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.852999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.852980    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7c7\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-kube-api-access-5k7c7\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.853051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.853021    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.853165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.853140    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.853222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.853179    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.853307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.853281    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.853357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.853331    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.953947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.953910    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.953971    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.953999    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954055    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954211    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7c7\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-kube-api-access-5k7c7\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954250    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954296    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954318    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954362    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954409    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954437    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954603    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954693    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.954928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.954906    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.956944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.956927    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.956995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.956971    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.974078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.974042    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:27.974204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:27.974167    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7c7\" (UniqueName: \"kubernetes.io/projected/a5c89cdf-dad9-4a02-8e82-347059ce2a4e-kube-api-access-5k7c7\") pod \"maas-default-gateway-openshift-default-694db55c87-lv28h\" (UID: \"a5c89cdf-dad9-4a02-8e82-347059ce2a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:28.168907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.168831    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:28.510790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.510760    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"]
Nov 03 20:18:28.511920 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:28.511899    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c89cdf_dad9_4a02_8e82_347059ce2a4e.slice/crio-ea91b7bf341cd269c0d3423f2cfd4587889fc9b00e6976e523b3e013d2eaf9c0 WatchSource:0}: Error finding container ea91b7bf341cd269c0d3423f2cfd4587889fc9b00e6976e523b3e013d2eaf9c0: Status 404 returned error can't find the container with id ea91b7bf341cd269c0d3423f2cfd4587889fc9b00e6976e523b3e013d2eaf9c0
Nov 03 20:18:28.515062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.515026    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:28.515139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.515097    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:28.515139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.515129    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:28.805402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.805312    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:28.805402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:28.805374    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:29.388893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:29.388858    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" event={"ID":"a5c89cdf-dad9-4a02-8e82-347059ce2a4e","Type":"ContainerStarted","Data":"a9bdf12952af37dc162189335e3506adeb9a7c6130995a66b0244e6237c75921"}
Nov 03 20:18:29.389132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:29.389117    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" event={"ID":"a5c89cdf-dad9-4a02-8e82-347059ce2a4e","Type":"ContainerStarted","Data":"ea91b7bf341cd269c0d3423f2cfd4587889fc9b00e6976e523b3e013d2eaf9c0"}
Nov 03 20:18:29.455558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:29.447784    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podStartSLOduration=2.447762399 podStartE2EDuration="2.447762399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:29.446013694 +0000 UTC m=+851.655029720" watchObservedRunningTime="2026-11-03 20:18:29.447762399 +0000 UTC m=+851.656778397"
Nov 03 20:18:29.803769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:29.803729    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:29.804167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:29.803795    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:30.171622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:30.170020    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:30.171622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:30.171600    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:30.171834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:30.171666    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:30.803345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:30.803306    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:30.803541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:30.803362    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:31.172614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:31.172098    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:31.172614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:31.172162    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:31.803896 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:31.803782    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:31.803896 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:31.803848    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:32.173490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:32.172961    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:32.173490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:32.173043    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:32.804147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:32.804108    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:32.804334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:32.804168    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:33.170107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:33.170014    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.170388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:33.170357    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:33.804092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:33.804047    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.804512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:33.804118    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:18:34.170026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:34.169934    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.170258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:34.170220    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:34.807976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:34.807942    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:34.808461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:34.808232    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:34.808947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:34.808931    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:18:35.169991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:35.169901    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:35.170257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:35.170227    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:36.169150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:36.169116    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:36.169654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:36.169169    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:18:37.197655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.197609    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:37.198573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.198548    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:37.199402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.199378    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h"
Nov 03 20:18:37.513549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.513499    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:18:37.517798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.517770    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.539692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.539664    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"distinct-w4-e80203-kserve-self-signed-certs\""
Nov 03 20:18:37.540582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.540562    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:18:37.587244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587207    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schlz\" (UniqueName: \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.587434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587268    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.587489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587426    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.587592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587575    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.587660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587605    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.587704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.587689    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688460    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688521    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688562    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688598    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688735    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schlz\" (UniqueName: \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.688903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688782    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.689016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.688983    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.689101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.689082    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.689204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.689179    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.691241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.691214    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.691383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.691365    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.742636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.740577    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schlz\" (UniqueName: \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\") pod \"distinct-w4-e80203-kserve-5cb7877787-8tfjf\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:37.841641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:37.841333    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:38.150002 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:38.149976    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf47ace9d_2dfe_4a97_8a32_9e00d230ed44.slice/crio-8557ce5633b18907ec359947f0619e5f4c84ecb9d15e9726bd260bb5e7aee640 WatchSource:0}: Error finding container 8557ce5633b18907ec359947f0619e5f4c84ecb9d15e9726bd260bb5e7aee640: Status 404 returned error can't find the container with id 8557ce5633b18907ec359947f0619e5f4c84ecb9d15e9726bd260bb5e7aee640
Nov 03 20:18:38.151187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:38.151162    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:18:38.453191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:38.453086    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" event={"ID":"f47ace9d-2dfe-4a97-8a32-9e00d230ed44","Type":"ContainerStarted","Data":"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"}
Nov 03 20:18:38.453191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:38.453141    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" event={"ID":"f47ace9d-2dfe-4a97-8a32-9e00d230ed44","Type":"ContainerStarted","Data":"8557ce5633b18907ec359947f0619e5f4c84ecb9d15e9726bd260bb5e7aee640"}
Nov 03 20:18:38.493049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:38.492994    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podStartSLOduration=1.492981544 podStartE2EDuration="1.492981544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:38.492416059 +0000 UTC m=+860.701432057" watchObservedRunningTime="2026-11-03 20:18:38.492981544 +0000 UTC m=+860.701997541"
Nov 03 20:18:38.842018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:38.841985    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:41.332369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.332332    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:18:41.336756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.336730    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.353763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.353729    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:18:41.361705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.361674    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"distinct-w2-9973d9-kserve-self-signed-certs\""
Nov 03 20:18:41.455550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455486    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.455759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455674    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.455759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455727    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.455889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455819    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28fh\" (UniqueName: \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.455889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455861    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.455993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.455925    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.490117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.490080    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:18:41.494286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.494263    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.512139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.512109    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e40816-openshift-default-dockercfg-rbqqp\""
Nov 03 20:18:41.525182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.521190    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:18:41.556902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.556831    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.556902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.556892    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnz7\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557050    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557083    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.557193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557109    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557133    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557183    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557268    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557309    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557337    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557387    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557427    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28fh\" (UniqueName: \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557455    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.557491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557491    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.558013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.557563    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.558095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.558064    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.559064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.559038    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.559376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.559341    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.561394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.561366    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.562622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.562602    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.599611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.590987    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28fh\" (UniqueName: \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\") pod \"distinct-w2-9973d9-kserve-dffd5b4bd-vk28n\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.656730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.656685    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:41.659479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659446    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659505    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659559    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnz7\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659661    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659690    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659718    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659771    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.659868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659866    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659889    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.659904    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.660214    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.660482    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.660792    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.660944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.660918    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.663452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.663410    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.664080 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.664053    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.682085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.682053    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnz7\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.682957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.682931    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.811039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.811008    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:41.985025 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:41.984989    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7acb85c4_bd6f_4041_8a15_ae289c527c26.slice/crio-391ce8f26510b913bdb23d24b4bf849f459bf31c206604a2f0561e7e6650215c WatchSource:0}: Error finding container 391ce8f26510b913bdb23d24b4bf849f459bf31c206604a2f0561e7e6650215c: Status 404 returned error can't find the container with id 391ce8f26510b913bdb23d24b4bf849f459bf31c206604a2f0561e7e6650215c
Nov 03 20:18:41.986015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:41.985987    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:18:42.117253 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:42.117132    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00eb9ba9_5be7_42f8_a745_63ec158d9942.slice/crio-07c288478da67fafb69634bee3d26e58a902e1a11d2066f42e976ff814253425 WatchSource:0}: Error finding container 07c288478da67fafb69634bee3d26e58a902e1a11d2066f42e976ff814253425: Status 404 returned error can't find the container with id 07c288478da67fafb69634bee3d26e58a902e1a11d2066f42e976ff814253425
Nov 03 20:18:42.117934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.117903    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:18:42.119843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.119810    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:42.119985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.119880    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:42.119985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.119912    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:42.290115 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:18:42.290074    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:18:42.503618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.503581    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" event={"ID":"00eb9ba9-5be7-42f8-a745-63ec158d9942","Type":"ContainerStarted","Data":"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"}
Nov 03 20:18:42.504088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.503646    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" event={"ID":"00eb9ba9-5be7-42f8-a745-63ec158d9942","Type":"ContainerStarted","Data":"07c288478da67fafb69634bee3d26e58a902e1a11d2066f42e976ff814253425"}
Nov 03 20:18:42.505595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.505573    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" event={"ID":"7acb85c4-bd6f-4041-8a15-ae289c527c26","Type":"ContainerStarted","Data":"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"}
Nov 03 20:18:42.505719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.505601    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" event={"ID":"7acb85c4-bd6f-4041-8a15-ae289c527c26","Type":"ContainerStarted","Data":"391ce8f26510b913bdb23d24b4bf849f459bf31c206604a2f0561e7e6650215c"}
Nov 03 20:18:42.545219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.545157    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podStartSLOduration=1.545142864 podStartE2EDuration="1.545142864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:42.542302621 +0000 UTC m=+864.751318628" watchObservedRunningTime="2026-11-03 20:18:42.545142864 +0000 UTC m=+864.754158861"
Nov 03 20:18:42.581649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.581601    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podStartSLOduration=1.581588849 podStartE2EDuration="1.581588849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:42.580817987 +0000 UTC m=+864.789833984" watchObservedRunningTime="2026-11-03 20:18:42.581588849 +0000 UTC m=+864.790604846"
Nov 03 20:18:42.657497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.657416    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:42.811942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:42.811893    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:43.814087 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:43.814050    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:43.814481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:43.814107    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:44.812341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:44.812305    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:44.812518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:44.812354    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:45.812108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:45.812075    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:45.812557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:45.812141    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:46.812030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:46.811987    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:46.812245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:46.812052    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:47.811680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:47.811644    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:47.811874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:47.811699    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:47.844556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:47.844208    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:47.858801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:47.858767    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:48.593590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:48.593560    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:18:48.811578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:48.811509    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:48.811755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:48.811582    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:18:49.815600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:49.815572    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:49.816089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:49.816074    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:49.816913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:49.816889    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:18:51.657862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:51.657829    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:51.670467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:51.670442    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:52.613349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:52.613320    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:18:55.816431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.816396    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:18:55.821970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.821938    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.844501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.844466    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:18:55.971637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.971589    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.972055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972033    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.972258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972238    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.972451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972435    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.972660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972643    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tnk\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.972808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972793    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.974331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.972943    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.974815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.974785    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:55.975095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:55.975076    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077198    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077305    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077348    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077418    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077466    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tnk\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.077614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077494    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.078325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077936    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.078325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.077993    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.078325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.078094    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.078793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.078757    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.079638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.079114    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.079787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.079317    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.079898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.079727    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.080007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.079929    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.082426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.082395    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.083829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.083804    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.099032 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.098948    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.100934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.100905    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tnk\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.140138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.140093    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:56.615559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.606159    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:18:56.658923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.658886    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783722    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783769    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783832    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783882    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783935    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.783968    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.784060    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.784127    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.794792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.784179    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvwh\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.825994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.825124    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:56.825994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.825229    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:56.825994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.825320    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:56.826824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.826749    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:18:56.860556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.847629    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885481    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885581    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885690    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885727    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885821    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885867    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.885919    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvwh\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.886087    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.886114    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.887614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.887361    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.888280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.887713    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.888280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.887939    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.888280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.888180    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.889384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.889361    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.893044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.893020    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.893702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.893680    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.917941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.917869    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:56.933175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:56.926211    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvwh\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-85qct\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:57.131090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.130994    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:57.534084 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:18:57.534047    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408b216c_7209_4321_9215_86d5cc0c6455.slice/crio-4e0ee0a9bc62b0f1f9f5cf9a45dbb4c2b4ad906a2b661049cb721c8f8f8d12c7 WatchSource:0}: Error finding container 4e0ee0a9bc62b0f1f9f5cf9a45dbb4c2b4ad906a2b661049cb721c8f8f8d12c7: Status 404 returned error can't find the container with id 4e0ee0a9bc62b0f1f9f5cf9a45dbb4c2b4ad906a2b661049cb721c8f8f8d12c7
Nov 03 20:18:57.535185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.535162    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:18:57.544565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.537546    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:57.544565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.537713    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:57.544565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.537829    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:57.759353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.759266    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" event={"ID":"3c910095-27d0-4a7f-87de-908bdb8b94c7","Type":"ContainerStarted","Data":"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"}
Nov 03 20:18:57.759561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.759438    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" event={"ID":"3c910095-27d0-4a7f-87de-908bdb8b94c7","Type":"ContainerStarted","Data":"bc49a9c123d54b353fcb61ccbc3ff6ed309b5a70c869655c84d75c52156e689a"}
Nov 03 20:18:57.761485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.761455    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" event={"ID":"408b216c-7209-4321-9215-86d5cc0c6455","Type":"ContainerStarted","Data":"2bde411639307235606954eb8d13aadbaacff4c11f1725585223580b1960a945"}
Nov 03 20:18:57.761646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.761504    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" event={"ID":"408b216c-7209-4321-9215-86d5cc0c6455","Type":"ContainerStarted","Data":"4e0ee0a9bc62b0f1f9f5cf9a45dbb4c2b4ad906a2b661049cb721c8f8f8d12c7"}
Nov 03 20:18:57.803174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:57.803062    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podStartSLOduration=2.803047534 podStartE2EDuration="2.803047534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:57.801859431 +0000 UTC m=+880.010875429" watchObservedRunningTime="2026-11-03 20:18:57.803047534 +0000 UTC m=+880.012063531"
Nov 03 20:18:58.136159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.134908    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:18:58.143579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.141663    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:18:58.143579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.141739    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:18:58.143579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.143574    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:18:58.143881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.143666    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:18:58.143881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:58.143706    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:18:59.141555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:59.140938    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:18:59.141555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:59.141007    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:18:59.145782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:59.145659    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:18:59.145782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:18:59.145725    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:00.136911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:00.135891    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:19:00.136911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:00.135954    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:00.153742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:00.150074    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:19:00.153742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:00.150138    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:19:01.136167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:01.136113    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:19:01.136474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:01.136420    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:01.144556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:01.140915    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:19:01.144556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:01.140964    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:19:02.088825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.088775    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podStartSLOduration=6.088761061 podStartE2EDuration="6.088761061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:57.843878442 +0000 UTC m=+880.052894441" watchObservedRunningTime="2026-11-03 20:19:02.088761061 +0000 UTC m=+884.297777058"
Nov 03 20:19:02.091779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.091758    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:19:02.096071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.096043    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.153557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.142150    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"distinct-w1-6fc748-kserve-self-signed-certs\""
Nov 03 20:19:02.153557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.142612    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:19:02.153557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.142676    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:02.158561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.158011    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:19:02.161692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.161621    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:19:02.187619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.183371    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:19:02.189490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.189451    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wwg\" (UniqueName: \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.189788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.189763    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.189922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.189906    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.190291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.190267    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.190382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.190364    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.190439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.190396    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294160    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294262    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294302    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294351    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wwg\" (UniqueName: \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294423    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294446    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.294753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.294726    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.296079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.296052    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.296254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.296143    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.299613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.297817    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.299613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.299407    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.318555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.318468    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wwg\" (UniqueName: \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\") pod \"distinct-w1-6fc748-kserve-67d4f574-pq8n5\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.414975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.414808    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:02.944087 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:19:02.944041    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f345309_2a86_4e03_b805_9089c9023b91.slice/crio-e728d330ffbab7a0dc3264cbbdac394e31eb08ded28a5fa3c18f93920eb5891e WatchSource:0}: Error finding container e728d330ffbab7a0dc3264cbbdac394e31eb08ded28a5fa3c18f93920eb5891e: Status 404 returned error can't find the container with id e728d330ffbab7a0dc3264cbbdac394e31eb08ded28a5fa3c18f93920eb5891e
Nov 03 20:19:02.963569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:02.962306    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:19:03.133027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.132977    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:19:03.133562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.133048    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:03.143079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.143014    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:19:03.143303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.143086    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:19:03.812899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.812851    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" event={"ID":"4f345309-2a86-4e03-b805-9089c9023b91","Type":"ContainerStarted","Data":"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"}
Nov 03 20:19:03.812899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.812895    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" event={"ID":"4f345309-2a86-4e03-b805-9089c9023b91","Type":"ContainerStarted","Data":"e728d330ffbab7a0dc3264cbbdac394e31eb08ded28a5fa3c18f93920eb5891e"}
Nov 03 20:19:03.853980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:03.853923    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podStartSLOduration=1.853905622 podStartE2EDuration="1.853905622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:03.850999389 +0000 UTC m=+886.060015387" watchObservedRunningTime="2026-11-03 20:19:03.853905622 +0000 UTC m=+886.062921620"
Nov 03 20:19:04.133422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.133330    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:19:04.133854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.133432    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:19:04.153329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.153294    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:19:04.154137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.154043    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:19:04.155252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.155231    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:19:04.418826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:04.418726    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:05.138222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:05.138192    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:19:05.138653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:05.138575    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:19:05.139360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:05.139343    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:19:11.676475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.676439    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:19:11.680705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.680682    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.703684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.703650    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:19:11.799360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799312    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsx8\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799446    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799487    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799519    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799560    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799699    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799749    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799790    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.799882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.799869    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901477    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901596    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsx8\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901652    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901691    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901732    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.901962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901889    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.901957    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902015    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902073    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902098    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902113    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902017    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902401    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.902623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.902602    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.904262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.904238    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.904358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.904310    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.921423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.921397    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.921927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.921898    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsx8\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:11.999220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:11.999186    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:12.288812 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:19:12.288732    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:19:12.323659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.323634    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:19:12.324320 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:19:12.324289    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66efb845_9ea2_422b_9be1_42ddfabeaf7b.slice/crio-f5f7a6f7d2d75ceaf26438d272a9283abc3ee8f8e5be0d64aea733ba465c0814 WatchSource:0}: Error finding container f5f7a6f7d2d75ceaf26438d272a9283abc3ee8f8e5be0d64aea733ba465c0814: Status 404 returned error can't find the container with id f5f7a6f7d2d75ceaf26438d272a9283abc3ee8f8e5be0d64aea733ba465c0814
Nov 03 20:19:12.327414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.327378    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:12.327498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.327473    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:12.327590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.327503    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:12.415696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.415652    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:12.430115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.430084    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:12.896512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.896480    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" event={"ID":"66efb845-9ea2-422b-9be1-42ddfabeaf7b","Type":"ContainerStarted","Data":"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"}
Nov 03 20:19:12.897021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.896564    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" event={"ID":"66efb845-9ea2-422b-9be1-42ddfabeaf7b","Type":"ContainerStarted","Data":"f5f7a6f7d2d75ceaf26438d272a9283abc3ee8f8e5be0d64aea733ba465c0814"}
Nov 03 20:19:12.913053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.913014    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:19:12.952605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:12.952519    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podStartSLOduration=1.952496381 podStartE2EDuration="1.952496381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:12.950916977 +0000 UTC m=+895.159932975" watchObservedRunningTime="2026-11-03 20:19:12.952496381 +0000 UTC m=+895.161512380"
Nov 03 20:19:13.000228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:13.000198    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:13.001639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:13.001600    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:13.001765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:13.001659    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:14.000695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:14.000661    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:14.001233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:14.000722    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:14.999621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:14.999583    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:14.999834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:14.999641    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:16.000560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:16.000494    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:16.001086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:16.000602    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:16.999839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:16.999802    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:17.000038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:16.999869    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:17.999638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:17.999594    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:18.000113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:17.999664    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:19.000074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:19.000025    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:19.000626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:19.000091    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:19:20.004668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:20.004639    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:20.005060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:20.004934    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:20.005594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:20.005575    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:19:26.696808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.696773    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:19:26.702296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.702274    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.731374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.731331    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:19:26.822820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.822777    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.823168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.823147    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.823395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.823381    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.823560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.823512    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.823770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.823744    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.823919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.823903    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5dx\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.824104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.824091    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.824363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.824227    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.824802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.824783    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926254    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926326    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926383    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926501    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926601    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926641    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926688    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5dx\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926790    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.926832    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.929445    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.930140    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.930249    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.930271    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.940550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.930759    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.943694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.943179    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.962862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.956324    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5dx\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.967582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.967502    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:26.972417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:26.972033    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-4pgk2\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:27.026600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:27.025730    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:27.583066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:27.582232    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:27.583066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:27.582331    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:27.583066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:27.582375    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:27.614011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:27.612713    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:19:28.082491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:28.082453    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" event={"ID":"52122ad3-9e3d-4c10-86f6-c4ccb9207740","Type":"ContainerStarted","Data":"216a8a645757492e37352cb95c764e5bfae55f79a8c6b05d6cea31fe25d6ad0b"}
Nov 03 20:19:28.083123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:28.083094    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" event={"ID":"52122ad3-9e3d-4c10-86f6-c4ccb9207740","Type":"ContainerStarted","Data":"6f25255aa329f54df6a31327fa507711982a822270e7e834e2d9d71ac573bbe8"}
Nov 03 20:19:28.310567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:28.245236    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podStartSLOduration=2.245209656 podStartE2EDuration="2.245209656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:28.24301015 +0000 UTC m=+910.452026184" watchObservedRunningTime="2026-11-03 20:19:28.245209656 +0000 UTC m=+910.454225654"
Nov 03 20:19:28.625295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:28.625194    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:28.893615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:28.888656    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:29.034565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:29.028875    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:29.034565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:29.031450    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:29.034565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:29.031509    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:29.063565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:29.062618    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:30.034556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:30.026215    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:30.034556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:30.026277    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:31.037955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:31.037914    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:31.038522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:31.038488    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:32.031966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:32.031927    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:32.032971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:32.032937    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:33.035561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:33.034909    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:33.035561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:33.034973    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:34.042557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:34.041941    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:34.042557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:34.042011    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:35.033751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:35.033712    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:19:35.034150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:35.034119    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 20:19:36.147706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:36.147669    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:36.148238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:36.148182    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:36.149637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:36.149615    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:19:37.187852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.187822    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:19:37.192275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.192245    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.209684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.209649    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"distinct-2-w2-9973d9-kserve-self-signed-certs\""
Nov 03 20:19:37.212809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.212775    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:19:37.344107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344074    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.344307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344135    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.344459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344415    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6g7\" (UniqueName: \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.344665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344643    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.344740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344704    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.344859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.344842    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.445991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.445889    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.446731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.446618    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.446928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.446775    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.447026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.446960    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6g7\" (UniqueName: \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.447125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.447107    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.447188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.447157    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.447303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.447289    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.447801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.447773    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.448035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.448013    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.450825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.450791    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.451112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.451095    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.466276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.466246    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6g7\" (UniqueName: \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\") pod \"distinct-2-w2-9973d9-kserve-68867b96c-skpv7\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") " pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.514605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.514569    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:37.840481 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:19:37.840433    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e3429f7_df22_46d3_917b_d71fa0cf5ee6.slice/crio-144a41be5019811f89e046effa7a042d38db2718bd6bfbc0c4f6cc506398b75d WatchSource:0}: Error finding container 144a41be5019811f89e046effa7a042d38db2718bd6bfbc0c4f6cc506398b75d: Status 404 returned error can't find the container with id 144a41be5019811f89e046effa7a042d38db2718bd6bfbc0c4f6cc506398b75d
Nov 03 20:19:37.841186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:37.841156    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:19:38.204821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:38.204728    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" event={"ID":"1e3429f7-df22-46d3-917b-d71fa0cf5ee6","Type":"ContainerStarted","Data":"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"}
Nov 03 20:19:38.204821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:38.204787    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" event={"ID":"1e3429f7-df22-46d3-917b-d71fa0cf5ee6","Type":"ContainerStarted","Data":"144a41be5019811f89e046effa7a042d38db2718bd6bfbc0c4f6cc506398b75d"}
Nov 03 20:19:38.246074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:38.246022    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podStartSLOduration=1.246006817 podStartE2EDuration="1.246006817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:38.242352889 +0000 UTC m=+920.451368886" watchObservedRunningTime="2026-11-03 20:19:38.246006817 +0000 UTC m=+920.455022813"
Nov 03 20:19:38.515679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:38.515645    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:42.285441 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:19:42.285400    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:19:44.576566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.565607    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:19:44.613302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.612638    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:44.733624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.731768    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:19:44.733624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.731912    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926465    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926620    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926703    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926764    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926812    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.927620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.926854    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5x78\" (UniqueName: \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:44.968557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:44.968505    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fd01ad\"/\"distinct-2-w5-fd01ad-kserve-self-signed-certs\""
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.057939    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.058050    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.058124    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.058180    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.058227    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5x78\" (UniqueName: \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.059555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.058471    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.060839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.060813    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.068683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.068643    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.095576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.069707    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.095576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.069982    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.163675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.163622    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.180348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.180224    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.183381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.183326    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5x78\" (UniqueName: \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.247553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.246287    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\") pod \"distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") " pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.360650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.360514    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:45.862685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:45.862638    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:19:46.369945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:46.369904    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" event={"ID":"a18071f0-b070-4d57-aa55-a6ae470f34b5","Type":"ContainerStarted","Data":"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"}
Nov 03 20:19:46.370129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:46.369955    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" event={"ID":"a18071f0-b070-4d57-aa55-a6ae470f34b5","Type":"ContainerStarted","Data":"56ece64f324234528521fedeb8e03c85d42789ca4487c4770d2c10ca55af97ea"}
Nov 03 20:19:46.411826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:46.411774    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podStartSLOduration=2.411759339 podStartE2EDuration="2.411759339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:46.409600211 +0000 UTC m=+928.618616208" watchObservedRunningTime="2026-11-03 20:19:46.411759339 +0000 UTC m=+928.620775368"
Nov 03 20:19:47.145164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.145132    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"]
Nov 03 20:19:47.149172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.149151    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.169849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.169820    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:19:47.181773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.181644    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:19:47.186438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.186367    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:19:47.186782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.186545    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:19:47.188637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.188602    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"]
Nov 03 20:19:47.209429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.209389    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.209628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.209509    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlbn\" (UniqueName: \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.209770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.209750    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.311568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.311507    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.311783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.311647    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.311847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.311798    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlbn\" (UniqueName: \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.312368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.312300    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.314339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.314312    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.332255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.332210    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlbn\" (UniqueName: \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\") pod \"maas-api-revoke-sub-fb8b2133-vws4b\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.361695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.361663    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:47.464602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.464504    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:47.518576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.515510    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:47.534930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.534905    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:47.774036 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:19:47.773992    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acb4e95_17c7_4acd_8825_c6654363b37c.slice/crio-ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc WatchSource:0}: Error finding container ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc: Status 404 returned error can't find the container with id ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc
Nov 03 20:19:47.775476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:47.775451    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"]
Nov 03 20:19:48.386097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:48.386001    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b" event={"ID":"4acb4e95-17c7-4acd-8825-c6654363b37c","Type":"ContainerStarted","Data":"d8fd864d474b28036103bddde8793d44c6c462b1a6df675916de6cdb5b1d166f"}
Nov 03 20:19:48.386097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:48.386062    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b" event={"ID":"4acb4e95-17c7-4acd-8825-c6654363b37c","Type":"ContainerStarted","Data":"ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc"}
Nov 03 20:19:48.401946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:48.401912    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:19:48.425383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:48.425328    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b" podStartSLOduration=1.425313804 podStartE2EDuration="1.425313804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:48.42398552 +0000 UTC m=+930.633001519" watchObservedRunningTime="2026-11-03 20:19:48.425313804 +0000 UTC m=+930.634329852"
Nov 03 20:19:49.396460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:49.396423    5424 generic.go:356] "Generic (PLEG): container finished" podID="4acb4e95-17c7-4acd-8825-c6654363b37c" containerID="d8fd864d474b28036103bddde8793d44c6c462b1a6df675916de6cdb5b1d166f" exitCode=0
Nov 03 20:19:49.396932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:49.396507    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b" event={"ID":"4acb4e95-17c7-4acd-8825-c6654363b37c","Type":"ContainerDied","Data":"d8fd864d474b28036103bddde8793d44c6c462b1a6df675916de6cdb5b1d166f"}
Nov 03 20:19:50.698952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.698876    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:50.781354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.781320    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\") pod \"4acb4e95-17c7-4acd-8825-c6654363b37c\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") "
Nov 03 20:19:50.781636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.781378    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\" (UniqueName: \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\") pod \"4acb4e95-17c7-4acd-8825-c6654363b37c\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") "
Nov 03 20:19:50.781636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.781437    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\") pod \"4acb4e95-17c7-4acd-8825-c6654363b37c\" (UID: \"4acb4e95-17c7-4acd-8825-c6654363b37c\") "
Nov 03 20:19:50.782287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.782251    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca" pod "4acb4e95-17c7-4acd-8825-c6654363b37c" (UID: "4acb4e95-17c7-4acd-8825-c6654363b37c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:50.783338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.783310    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:19:50.787689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.787660    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca" pod "4acb4e95-17c7-4acd-8825-c6654363b37c" (UID: "4acb4e95-17c7-4acd-8825-c6654363b37c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:50.790541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.790498    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn" pod "4acb4e95-17c7-4acd-8825-c6654363b37c" (UID: "4acb4e95-17c7-4acd-8825-c6654363b37c"). InnerVolumeSpecName "kube-api-access-pmlbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:50.884424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.884379    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlbn\" (UniqueName: \"kubernetes.io/projected/4acb4e95-17c7-4acd-8825-c6654363b37c-kube-api-access-pmlbn\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:19:50.884424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:50.884418    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acb4e95-17c7-4acd-8825-c6654363b37c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:19:51.414186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:51.414155    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"
Nov 03 20:19:51.414186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:51.414181    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b" event={"ID":"4acb4e95-17c7-4acd-8825-c6654363b37c","Type":"ContainerDied","Data":"ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc"}
Nov 03 20:19:51.414387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:51.414230    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace5d6f0f8d9f6f1645bef7b25aa53875a5704862afff4b0a6a4024718d04bdc"
Nov 03 20:19:54.958947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.958908    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:19:54.959842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.959819    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4acb4e95-17c7-4acd-8825-c6654363b37c" containerName="revoke-keys"
Nov 03 20:19:54.959902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.959846    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acb4e95-17c7-4acd-8825-c6654363b37c" containerName="revoke-keys"
Nov 03 20:19:54.960075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.960060    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acb4e95-17c7-4acd-8825-c6654363b37c" containerName="revoke-keys"
Nov 03 20:19:54.964237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.964213    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:54.981958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.981899    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:19:54.989026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:54.988999    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"distinct-2-w1-6fc748-kserve-self-signed-certs\""
Nov 03 20:19:55.041593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.041557    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.041593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.041596    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.041820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.041618    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.041869 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.041840    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9p4n\" (UniqueName: \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.041920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.041897    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.042044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.042025    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144054    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144107    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144134    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144295    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9p4n\" (UniqueName: \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144342    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.144423    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.145052    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.145615    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.146568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.145837    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.167592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.161360    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.201565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.197281    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9p4n\" (UniqueName: \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.201565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.197505    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\") pod \"distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") " pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.281766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.281725    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:19:55.383974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.373726    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:55.422617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.420831    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:55.483377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.479802    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:19:55.606483 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:19:55.606453    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb11a3fd_7705_4e94_9345_9e22c8bff5f2.slice/crio-caec19c164d6c5a55a20d75d556566f54271acc0c6f423343d18205a5ad3ba65 WatchSource:0}: Error finding container caec19c164d6c5a55a20d75d556566f54271acc0c6f423343d18205a5ad3ba65: Status 404 returned error can't find the container with id caec19c164d6c5a55a20d75d556566f54271acc0c6f423343d18205a5ad3ba65
Nov 03 20:19:55.610729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:55.610702    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:19:56.471630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:56.471586    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" event={"ID":"db11a3fd-7705-4e94-9345-9e22c8bff5f2","Type":"ContainerStarted","Data":"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"}
Nov 03 20:19:56.472108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:56.471638    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" event={"ID":"db11a3fd-7705-4e94-9345-9e22c8bff5f2","Type":"ContainerStarted","Data":"caec19c164d6c5a55a20d75d556566f54271acc0c6f423343d18205a5ad3ba65"}
Nov 03 20:19:56.511384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:56.511332    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podStartSLOduration=2.511317582 podStartE2EDuration="2.511317582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:56.510492797 +0000 UTC m=+938.719508794" watchObservedRunningTime="2026-11-03 20:19:56.511317582 +0000 UTC m=+938.720333578"
Nov 03 20:19:57.282521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:19:57.282492    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:01.247893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.247852    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"]
Nov 03 20:20:01.252651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.252626    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.268606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.268579    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"]
Nov 03 20:20:01.276969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.276872    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:01.276969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.276891    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:01.276969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.276937    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:01.280202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.280176    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:01.437043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.437004    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.437282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.437262    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.437486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.437462    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9ct\" (UniqueName: \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.539279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.539198    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.539442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.539301    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9ct\" (UniqueName: \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.539442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.539355    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.539883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.539858    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.541962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.541938    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.561335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.561304    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9ct\" (UniqueName: \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\") pod \"maas-api-revoke-sub-82271bb2-dqnss\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.571231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.571204    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:01.879595 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:01.879564    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69aa372d_e9ff_4dfe_90de_7c24e9f4908e.slice/crio-9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661 WatchSource:0}: Error finding container 9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661: Status 404 returned error can't find the container with id 9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661
Nov 03 20:20:01.880113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:01.880093    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"]
Nov 03 20:20:02.513583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:02.513550    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss" event={"ID":"69aa372d-e9ff-4dfe-90de-7c24e9f4908e","Type":"ContainerStarted","Data":"ca3e436af811f23510503b6983c9b4d540152e737b6395c5f5e4fe85414c4d8c"}
Nov 03 20:20:02.514006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:02.513632    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss" event={"ID":"69aa372d-e9ff-4dfe-90de-7c24e9f4908e","Type":"ContainerStarted","Data":"9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661"}
Nov 03 20:20:02.554357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:02.554309    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss" podStartSLOduration=1.554295857 podStartE2EDuration="1.554295857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:02.552826497 +0000 UTC m=+944.761842494" watchObservedRunningTime="2026-11-03 20:20:02.554295857 +0000 UTC m=+944.763312278"
Nov 03 20:20:03.522847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:03.522809    5424 generic.go:356] "Generic (PLEG): container finished" podID="69aa372d-e9ff-4dfe-90de-7c24e9f4908e" containerID="ca3e436af811f23510503b6983c9b4d540152e737b6395c5f5e4fe85414c4d8c" exitCode=0
Nov 03 20:20:03.523248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:03.522872    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss" event={"ID":"69aa372d-e9ff-4dfe-90de-7c24e9f4908e","Type":"ContainerDied","Data":"ca3e436af811f23510503b6983c9b4d540152e737b6395c5f5e4fe85414c4d8c"}
Nov 03 20:20:04.684669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.684634    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:04.781100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.781001    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\") pod \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") "
Nov 03 20:20:04.781100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.781050    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\" (UniqueName: \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\") pod \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") "
Nov 03 20:20:04.781343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.781275    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\") pod \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\" (UID: \"69aa372d-e9ff-4dfe-90de-7c24e9f4908e\") "
Nov 03 20:20:04.781400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.781366    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca" pod "69aa372d-e9ff-4dfe-90de-7c24e9f4908e" (UID: "69aa372d-e9ff-4dfe-90de-7c24e9f4908e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:04.781837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.781816    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:04.783392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.783362    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct" pod "69aa372d-e9ff-4dfe-90de-7c24e9f4908e" (UID: "69aa372d-e9ff-4dfe-90de-7c24e9f4908e"). InnerVolumeSpecName "kube-api-access-lh9ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:04.783497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.783477    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca" pod "69aa372d-e9ff-4dfe-90de-7c24e9f4908e" (UID: "69aa372d-e9ff-4dfe-90de-7c24e9f4908e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:04.883068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.883032    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:04.883068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:04.883062    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9ct\" (UniqueName: \"kubernetes.io/projected/69aa372d-e9ff-4dfe-90de-7c24e9f4908e-kube-api-access-lh9ct\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:05.115702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.115611    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"]
Nov 03 20:20:05.116775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.116745    5424 cpu_manager.go:414] "containerMap: removing container" podUID="69aa372d-e9ff-4dfe-90de-7c24e9f4908e" containerName="revoke-keys"
Nov 03 20:20:05.116962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.116937    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69aa372d-e9ff-4dfe-90de-7c24e9f4908e" containerName="revoke-keys"
Nov 03 20:20:05.117143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.117128    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="69aa372d-e9ff-4dfe-90de-7c24e9f4908e" containerName="revoke-keys"
Nov 03 20:20:05.122305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.122280    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.148777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.139424    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"]
Nov 03 20:20:05.187751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.187719    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.187994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.187765    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.187994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.187957    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p44\" (UniqueName: \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.282489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.282452    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:05.290407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.289594    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.290407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.289651    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.290407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.289770    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p44\" (UniqueName: \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.291108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.291083    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.299136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.299100    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:05.309372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.308020    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.325228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.325192    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p44\" (UniqueName: \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\") pod \"maas-api-revoke-sub-fa0f0c22-w25v6\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.479444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.479346    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:05.626921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.626254    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"
Nov 03 20:20:05.627206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.627179    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss" event={"ID":"69aa372d-e9ff-4dfe-90de-7c24e9f4908e","Type":"ContainerDied","Data":"9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661"}
Nov 03 20:20:05.627317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.627235    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d81850196e3925cee8921e660b02f0603f44b7921b35ed18320b1e667050661"
Nov 03 20:20:05.687573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:05.687502    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:06.383670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:06.374348    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"]
Nov 03 20:20:06.690555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:06.689623    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6" event={"ID":"7ceafee3-ff21-434e-8bd0-85e17341e648","Type":"ContainerStarted","Data":"e957cfcac88eb14425572859c04ac2e5e68d4f466d2bb7260edfa19fb9b267eb"}
Nov 03 20:20:07.047558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.046633    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.195159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.194763    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:07.195159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.194827    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.196329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.196303    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.325571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.317819    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.474721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.474617    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.517189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.516659    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.518622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.518596    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.715762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.715722    5424 generic.go:356] "Generic (PLEG): container finished" podID="7ceafee3-ff21-434e-8bd0-85e17341e648" containerID="ac2362c76ede00f915c872c0cca44d924db4c25fc195cfaf11971167e007e72d" exitCode=0
Nov 03 20:20:07.716452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.716423    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6" event={"ID":"7ceafee3-ff21-434e-8bd0-85e17341e648","Type":"ContainerDied","Data":"ac2362c76ede00f915c872c0cca44d924db4c25fc195cfaf11971167e007e72d"}
Nov 03 20:20:07.908297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:07.907686    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.092576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.037688    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.092576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.085915    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.092576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.087335    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.138397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.138344    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.223151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.223020    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:08.420884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.420855    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:08.520685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.520637    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Nov 03 20:20:08.605555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:08.605396    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.316841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:09.316779    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.421691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:09.421632    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.482557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:09.482018    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.848600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:09.760675    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.058616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.058260    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.215556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.209392    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:10.303774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.299969    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\") pod \"7ceafee3-ff21-434e-8bd0-85e17341e648\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") "
Nov 03 20:20:10.303774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.300073    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\" (UniqueName: \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\") pod \"7ceafee3-ff21-434e-8bd0-85e17341e648\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") "
Nov 03 20:20:10.304555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.304485    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\") pod \"7ceafee3-ff21-434e-8bd0-85e17341e648\" (UID: \"7ceafee3-ff21-434e-8bd0-85e17341e648\") "
Nov 03 20:20:10.306207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.306182    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca" pod "7ceafee3-ff21-434e-8bd0-85e17341e648" (UID: "7ceafee3-ff21-434e-8bd0-85e17341e648"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:10.347778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.347693    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca" pod "7ceafee3-ff21-434e-8bd0-85e17341e648" (UID: "7ceafee3-ff21-434e-8bd0-85e17341e648"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:10.349343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.349316    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44" pod "7ceafee3-ff21-434e-8bd0-85e17341e648" (UID: "7ceafee3-ff21-434e-8bd0-85e17341e648"). InnerVolumeSpecName "kube-api-access-m9p44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:10.434557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.409744    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:10.434846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.434823    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p44\" (UniqueName: \"kubernetes.io/projected/7ceafee3-ff21-434e-8bd0-85e17341e648-kube-api-access-m9p44\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:10.434965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.434951    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceafee3-ff21-434e-8bd0-85e17341e648-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:10.490841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.490796    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:10.491618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.491596    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:10.705182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.705094    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:10.932718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.932673    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6" event={"ID":"7ceafee3-ff21-434e-8bd0-85e17341e648","Type":"ContainerDied","Data":"e957cfcac88eb14425572859c04ac2e5e68d4f466d2bb7260edfa19fb9b267eb"}
Nov 03 20:20:10.932718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.932698    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"
Nov 03 20:20:10.932941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:10.932733    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e957cfcac88eb14425572859c04ac2e5e68d4f466d2bb7260edfa19fb9b267eb"
Nov 03 20:20:12.295807 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:20:12.295766    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:20:15.279489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.279454    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"]
Nov 03 20:20:15.280004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.279989    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7ceafee3-ff21-434e-8bd0-85e17341e648" containerName="revoke-keys"
Nov 03 20:20:15.280047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.280005    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceafee3-ff21-434e-8bd0-85e17341e648" containerName="revoke-keys"
Nov 03 20:20:15.280108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.280099    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceafee3-ff21-434e-8bd0-85e17341e648" containerName="revoke-keys"
Nov 03 20:20:15.284146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.284127    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.302176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.302154    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:15.302627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.302605    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:15.315056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.315031    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:15.315056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.315036    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:15.324909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.324884    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"]
Nov 03 20:20:15.409667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.409637    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dj6s\" (UniqueName: \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.409846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.409832    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.409962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.409945    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.511347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.511311    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.511566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.511412    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.511566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.511508    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dj6s\" (UniqueName: \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.512264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.512239    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.513844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.513824    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.533265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.533208    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dj6s\" (UniqueName: \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\") pod \"maas-api-revoke-sub-9844baaf-lswrb\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.599386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.599358    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:15.903864 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:15.903835    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e13f9b_b53c_4f2e_9d4d_5fe02dac08c9.slice/crio-ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5 WatchSource:0}: Error finding container ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5: Status 404 returned error can't find the container with id ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5
Nov 03 20:20:15.905332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.905304    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"]
Nov 03 20:20:15.973008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:15.972976    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb" event={"ID":"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9","Type":"ContainerStarted","Data":"ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5"}
Nov 03 20:20:16.985539 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:16.985486    5424 generic.go:356] "Generic (PLEG): container finished" podID="c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" containerID="985bffde5d57536c2029b61c159497866733cba362fdf270c3618f92488d7088" exitCode=0
Nov 03 20:20:16.985964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:16.985628    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb" event={"ID":"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9","Type":"ContainerDied","Data":"985bffde5d57536c2029b61c159497866733cba362fdf270c3618f92488d7088"}
Nov 03 20:20:18.154488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.154459    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:18.160637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.160614    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\") pod \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") "
Nov 03 20:20:18.160770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.160719    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\" (UniqueName: \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\") pod \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") "
Nov 03 20:20:18.160856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.160827    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\") pod \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\" (UID: \"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9\") "
Nov 03 20:20:18.161416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.161386    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca" pod "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" (UID: "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:18.161565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.161548    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:18.163075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.163049    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca" pod "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" (UID: "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:18.163282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.163257    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s" pod "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" (UID: "c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9"). InnerVolumeSpecName "kube-api-access-8dj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:18.262962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.262888    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:18.262962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.262916    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dj6s\" (UniqueName: \"kubernetes.io/projected/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9-kube-api-access-8dj6s\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:18.615700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.615624    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"]
Nov 03 20:20:18.616176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.616163    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" containerName="revoke-keys"
Nov 03 20:20:18.616248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.616177    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" containerName="revoke-keys"
Nov 03 20:20:18.616306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.616272    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" containerName="revoke-keys"
Nov 03 20:20:18.619838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.619821    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.635019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.634990    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"]
Nov 03 20:20:18.667800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.667766    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.667800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.667806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmks\" (UniqueName: \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.668045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.667853    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.769214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.769178    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.769214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.769218    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmks\" (UniqueName: \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.769460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.769268    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.770044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.770021    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.771780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.771761    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.788963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.788940    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmks\" (UniqueName: \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\") pod \"maas-api-revoke-sub-309f61ec-q9dz9\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:18.936963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:18.936860    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:19.005438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:19.005402    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb" event={"ID":"c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9","Type":"ContainerDied","Data":"ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5"}
Nov 03 20:20:19.005689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:19.005664    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf38c3ead4cf19bddb7a0c33699287700ef065b18c9b4136a0faf797e03dae5"
Nov 03 20:20:19.005862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:19.005849    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"
Nov 03 20:20:19.248268 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:19.248230    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fcd831_ce34_4eaa_a781_e226ec917203.slice/crio-eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd WatchSource:0}: Error finding container eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd: Status 404 returned error can't find the container with id eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd
Nov 03 20:20:19.250996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:19.250974    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"]
Nov 03 20:20:20.014118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:20.014078    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9" event={"ID":"01fcd831-ce34-4eaa-a781-e226ec917203","Type":"ContainerStarted","Data":"869073651d21f9a82a00fbc9d247eb6826a5563666b524ba1e56bca8436ce02c"}
Nov 03 20:20:20.014279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:20.014134    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9" event={"ID":"01fcd831-ce34-4eaa-a781-e226ec917203","Type":"ContainerStarted","Data":"eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd"}
Nov 03 20:20:20.054371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:20.054310    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9" podStartSLOduration=2.054290891 podStartE2EDuration="2.054290891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:20.052774555 +0000 UTC m=+962.261790551" watchObservedRunningTime="2026-11-03 20:20:20.054290891 +0000 UTC m=+962.263306893"
Nov 03 20:20:21.026973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.026939    5424 generic.go:356] "Generic (PLEG): container finished" podID="01fcd831-ce34-4eaa-a781-e226ec917203" containerID="869073651d21f9a82a00fbc9d247eb6826a5563666b524ba1e56bca8436ce02c" exitCode=0
Nov 03 20:20:21.027384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.027071    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9" event={"ID":"01fcd831-ce34-4eaa-a781-e226ec917203","Type":"ContainerDied","Data":"869073651d21f9a82a00fbc9d247eb6826a5563666b524ba1e56bca8436ce02c"}
Nov 03 20:20:21.507345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.507309    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"]
Nov 03 20:20:21.511494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.511474    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.526518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.526495    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"]
Nov 03 20:20:21.610118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.610077    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9c8\" (UniqueName: \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.610312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.610223    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.610312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.610256    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.711373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.711330    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.711603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.711380    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.711670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.711644    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9c8\" (UniqueName: \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.712215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.712141    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.714002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.713980    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.730742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.730715    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9c8\" (UniqueName: \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\") pod \"maas-api-revoke-sub-079b4967-mn55l\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:21.827154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:21.827074    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:22.136796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.136471    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"]
Nov 03 20:20:22.138163 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:20:22.137667    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95216b61_05d8_4878_a08d_16ce63a3520a.slice/crio-7546c6349e0478c7ac32600a5fc219a6d8f2f63ba34da32bcf70b2dcc3170592\": RecentStats: unable to find data in memory cache]"
Nov 03 20:20:22.183947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.183923    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:22.220347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.220308    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\") pod \"01fcd831-ce34-4eaa-a781-e226ec917203\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") "
Nov 03 20:20:22.220547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.220363    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\") pod \"01fcd831-ce34-4eaa-a781-e226ec917203\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") "
Nov 03 20:20:22.220547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.220474    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\" (UniqueName: \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\") pod \"01fcd831-ce34-4eaa-a781-e226ec917203\" (UID: \"01fcd831-ce34-4eaa-a781-e226ec917203\") "
Nov 03 20:20:22.221277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.221062    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca" pod "01fcd831-ce34-4eaa-a781-e226ec917203" (UID: "01fcd831-ce34-4eaa-a781-e226ec917203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:22.221277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.221228    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:22.224813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.224611    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca" pod "01fcd831-ce34-4eaa-a781-e226ec917203" (UID: "01fcd831-ce34-4eaa-a781-e226ec917203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:22.224813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.224735    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks" pod "01fcd831-ce34-4eaa-a781-e226ec917203" (UID: "01fcd831-ce34-4eaa-a781-e226ec917203"). InnerVolumeSpecName "kube-api-access-lwmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:22.322502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.322465    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fcd831-ce34-4eaa-a781-e226ec917203-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:22.322502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:22.322504    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmks\" (UniqueName: \"kubernetes.io/projected/01fcd831-ce34-4eaa-a781-e226ec917203-kube-api-access-lwmks\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:23.043442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.043387    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l" event={"ID":"95216b61-05d8-4878-a08d-16ce63a3520a","Type":"ContainerStarted","Data":"6e152415c9dc37e62228509f6cde5e2e2cab92bf5593c12a7e5b524f732ef10c"}
Nov 03 20:20:23.043675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.043449    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l" event={"ID":"95216b61-05d8-4878-a08d-16ce63a3520a","Type":"ContainerStarted","Data":"7546c6349e0478c7ac32600a5fc219a6d8f2f63ba34da32bcf70b2dcc3170592"}
Nov 03 20:20:23.045095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.045070    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9" event={"ID":"01fcd831-ce34-4eaa-a781-e226ec917203","Type":"ContainerDied","Data":"eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd"}
Nov 03 20:20:23.045095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.045086    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"
Nov 03 20:20:23.045223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.045112    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec80a76c817ab51881c8297f02332b60530292e52486cd1b6c5f306674d4fdd"
Nov 03 20:20:23.091992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:23.091938    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l" podStartSLOduration=2.091924721 podStartE2EDuration="2.091924721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:23.088105244 +0000 UTC m=+965.297121241" watchObservedRunningTime="2026-11-03 20:20:23.091924721 +0000 UTC m=+965.300940782"
Nov 03 20:20:24.054194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:24.054161    5424 generic.go:356] "Generic (PLEG): container finished" podID="95216b61-05d8-4878-a08d-16ce63a3520a" containerID="6e152415c9dc37e62228509f6cde5e2e2cab92bf5593c12a7e5b524f732ef10c" exitCode=0
Nov 03 20:20:24.054594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:24.054253    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l" event={"ID":"95216b61-05d8-4878-a08d-16ce63a3520a","Type":"ContainerDied","Data":"6e152415c9dc37e62228509f6cde5e2e2cab92bf5593c12a7e5b524f732ef10c"}
Nov 03 20:20:25.225393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.225370    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:25.262264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.262233    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\" (UniqueName: \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\") pod \"95216b61-05d8-4878-a08d-16ce63a3520a\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") "
Nov 03 20:20:25.262440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.262303    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\") pod \"95216b61-05d8-4878-a08d-16ce63a3520a\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") "
Nov 03 20:20:25.262440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.262353    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\") pod \"95216b61-05d8-4878-a08d-16ce63a3520a\" (UID: \"95216b61-05d8-4878-a08d-16ce63a3520a\") "
Nov 03 20:20:25.262952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.262922    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca" pod "95216b61-05d8-4878-a08d-16ce63a3520a" (UID: "95216b61-05d8-4878-a08d-16ce63a3520a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:25.263316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.263292    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:25.264879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.264854    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca" pod "95216b61-05d8-4878-a08d-16ce63a3520a" (UID: "95216b61-05d8-4878-a08d-16ce63a3520a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:25.264961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.264902    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8" pod "95216b61-05d8-4878-a08d-16ce63a3520a" (UID: "95216b61-05d8-4878-a08d-16ce63a3520a"). InnerVolumeSpecName "kube-api-access-dm9c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:25.365215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.365106    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9c8\" (UniqueName: \"kubernetes.io/projected/95216b61-05d8-4878-a08d-16ce63a3520a-kube-api-access-dm9c8\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:25.365215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:25.365148    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95216b61-05d8-4878-a08d-16ce63a3520a-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:26.069551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.069505    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l" event={"ID":"95216b61-05d8-4878-a08d-16ce63a3520a","Type":"ContainerDied","Data":"7546c6349e0478c7ac32600a5fc219a6d8f2f63ba34da32bcf70b2dcc3170592"}
Nov 03 20:20:26.069724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.069589    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7546c6349e0478c7ac32600a5fc219a6d8f2f63ba34da32bcf70b2dcc3170592"
Nov 03 20:20:26.069724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.069519    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"
Nov 03 20:20:26.106286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.106252    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:20:26.106928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.106910    5424 cpu_manager.go:414] "containerMap: removing container" podUID="01fcd831-ce34-4eaa-a781-e226ec917203" containerName="revoke-keys"
Nov 03 20:20:26.106928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.106929    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fcd831-ce34-4eaa-a781-e226ec917203" containerName="revoke-keys"
Nov 03 20:20:26.107057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.106937    5424 cpu_manager.go:414] "containerMap: removing container" podUID="95216b61-05d8-4878-a08d-16ce63a3520a" containerName="revoke-keys"
Nov 03 20:20:26.107057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.106942    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95216b61-05d8-4878-a08d-16ce63a3520a" containerName="revoke-keys"
Nov 03 20:20:26.107057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.107034    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fcd831-ce34-4eaa-a781-e226ec917203" containerName="revoke-keys"
Nov 03 20:20:26.107057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.107042    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="95216b61-05d8-4878-a08d-16ce63a3520a" containerName="revoke-keys"
Nov 03 20:20:26.111065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.111036    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.136556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.136485    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:20:26.174509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174477    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174517    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174625    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174738    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174760    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174826    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174894    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvj4\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.174977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174966    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.175008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.174992    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.276568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276515    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276590    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276632    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276716    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276742    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276793    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276890    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvj4\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.276961    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.277000    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.277636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.277605    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.285145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.277695    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.285145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.277882    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.285145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.278084    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.285145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.280404    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.285145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.280999    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.289192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.285709    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.296683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.296649    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.297148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.297122    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvj4\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.434797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.434769    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:26.517260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.517223    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:20:26.522581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.522559    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.552182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.552154    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:20:26.566218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.566184    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:20:26.570934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.570904    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.580731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.580640    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:20:26.581913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.581877    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.581962    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582024    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582088    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582142    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582236    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582329    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582402    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.582502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.582488    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rv8p\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.683587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683508    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.683801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683623    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.683801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683657    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rv8p\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.683801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683678    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.683801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683713    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683824    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683869    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rbb\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683898    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.683973    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684010    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684109    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684148    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684246    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684410    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684493    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684573    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684630    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684692    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.684829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684801    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684871    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.684938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.684915    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.685125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.685106    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.685196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.685168    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.686299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.686273    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.687175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.687154    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.703047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.703018    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.703713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.703692    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rv8p\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.755664 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:26.755633    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bc23ca_27dd_44cc_936a_5372b991894b.slice/crio-a90f6e66ea974002b452821b38f87e61e0c054dd2e5c055c3fdb8fc6e3573c98 WatchSource:0}: Error finding container a90f6e66ea974002b452821b38f87e61e0c054dd2e5c055c3fdb8fc6e3573c98: Status 404 returned error can't find the container with id a90f6e66ea974002b452821b38f87e61e0c054dd2e5c055c3fdb8fc6e3573c98
Nov 03 20:20:26.758350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.758326    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:20:26.758783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.758752    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:26.758831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.758819    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:26.758868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.758853    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:26.786037 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.785972    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786076    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786106    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786126    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786166    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rbb\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786198    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786258    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786369    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786427    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786435    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786368    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786661    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786703    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.786864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.786832    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.788872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.788844    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.788974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.788890    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.808855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.808822    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.808976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.808926    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rbb\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.841996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.841966    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:26.896385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.896300    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:26.945744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.945704    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:20:26.952462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.952431    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.991582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.981962    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:20:26.991582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.984190    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:20:26.992503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992467    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbl5\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.992670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992590    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.992731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992688    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.992782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992739    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992830    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992892    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.992932    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.993727    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.993076    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:26.997486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:26.996442    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.013248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.011482    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:20:27.080217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.080102    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" event={"ID":"e8bc23ca-27dd-44cc-936a-5372b991894b","Type":"ContainerStarted","Data":"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"}
Nov 03 20:20:27.080217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.080169    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" event={"ID":"e8bc23ca-27dd-44cc-936a-5372b991894b","Type":"ContainerStarted","Data":"a90f6e66ea974002b452821b38f87e61e0c054dd2e5c055c3fdb8fc6e3573c98"}
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098147    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098218    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098248    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098273    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098320    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098360    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098439    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098550    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098641    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbl5\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098701    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098761    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098836    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.099111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.098842    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099204    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099442    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099634    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099688    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099751    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jn5\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099793    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099820    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.099885    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.100817    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.106237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.101440    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.108119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.108088    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.108596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.108570    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.118406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.118381    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbl5\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.119756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.119730    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.128036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.127988    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podStartSLOduration=1.127972998 podStartE2EDuration="1.127972998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:27.125789726 +0000 UTC m=+969.334805726" watchObservedRunningTime="2026-11-03 20:20:27.127972998 +0000 UTC m=+969.336988995"
Nov 03 20:20:27.166076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.166042    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:20:27.166467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.166429    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.166699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.166674    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.166906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.166869    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.201078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201045    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201160    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201279    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jn5\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201318    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201416    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201448    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201503    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201602    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.201676 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201650    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.202008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.201966    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.202368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.202343    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.202482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.202427    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.202999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.202928    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.202999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.202971    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.204888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.204864    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.205642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.205620    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.222958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.222933    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.223079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.223053    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jn5\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.267854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.267820    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:20:27.268090 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:27.268060    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff7ffa7_fdd6_4dd4_bcdf_a14e098d933e.slice/crio-ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d WatchSource:0}: Error finding container ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d: Status 404 returned error can't find the container with id ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d
Nov 03 20:20:27.271446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.271407    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.271610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.271503    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.271610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.271571    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.295124 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.294854    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:27.325896 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.325620    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:27.452486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.452404    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:27.465469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.465120    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:27.465469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.465176    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:27.670508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.670477    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:20:27.674310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.672078    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.674310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.672162    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.674310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.672207    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.764185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.763816    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:20:27.767381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.767331    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.767737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.767693    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:27.767934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:27.767913    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:28.104342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.104299    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" event={"ID":"0cba4598-49ff-49c1-9afb-072fcce11c09","Type":"ContainerStarted","Data":"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"}
Nov 03 20:20:28.104504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.104368    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" event={"ID":"0cba4598-49ff-49c1-9afb-072fcce11c09","Type":"ContainerStarted","Data":"0e5c38c9ff9a0eda2b9defed620765d108ee8ba4952f92e89380ab54854bfa4f"}
Nov 03 20:20:28.111248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.111213    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" event={"ID":"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e","Type":"ContainerStarted","Data":"a94b660f25291fab9073817b2593a2dd19ea2b068f2bc9eedb51c12b50d4b203"}
Nov 03 20:20:28.111374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.111276    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" event={"ID":"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e","Type":"ContainerStarted","Data":"ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d"}
Nov 03 20:20:28.115207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.115166    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" event={"ID":"c92715e3-5479-4173-86af-e8d73bbb1570","Type":"ContainerStarted","Data":"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"}
Nov 03 20:20:28.115344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.115216    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" event={"ID":"c92715e3-5479-4173-86af-e8d73bbb1570","Type":"ContainerStarted","Data":"6694843f00f8b5cd477785124609696d972f5b26d6bab8b8e4a3bbfe4c336f5d"}
Nov 03 20:20:28.117144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.117119    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" event={"ID":"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4","Type":"ContainerStarted","Data":"6898270d2c98a5d63c3b2abb6482cb3ef05bb8f2d1b142dba1de0e14d5e87012"}
Nov 03 20:20:28.176556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.160969    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podStartSLOduration=2.1609517289999998 podStartE2EDuration="2.160951729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.159080505 +0000 UTC m=+970.368096545" watchObservedRunningTime="2026-11-03 20:20:28.160951729 +0000 UTC m=+970.369967726"
Nov 03 20:20:28.248547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.248474    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podStartSLOduration=2.248441016 podStartE2EDuration="2.248441016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.214859065 +0000 UTC m=+970.423875098" watchObservedRunningTime="2026-11-03 20:20:28.248441016 +0000 UTC m=+970.457457014"
Nov 03 20:20:28.297709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.297621    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:28.298300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.298276    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.298467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.298442    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:28.319660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.319597    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podStartSLOduration=2.319576817 podStartE2EDuration="2.319576817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.260661584 +0000 UTC m=+970.469677584" watchObservedRunningTime="2026-11-03 20:20:28.319576817 +0000 UTC m=+970.528592815"
Nov 03 20:20:28.331703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.331667    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:28.353554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.333870    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podStartSLOduration=2.333852377 podStartE2EDuration="2.333852377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.314585448 +0000 UTC m=+970.523601447" watchObservedRunningTime="2026-11-03 20:20:28.333852377 +0000 UTC m=+970.542868375"
Nov 03 20:20:28.442589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.435946    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.442589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.436006    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:28.848924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.848891    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:28.849245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.849173    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.849401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.849381    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:28.906009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.905981    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:28.906334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.906315    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.906469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:28.906449    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:29.186637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.182694    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" event={"ID":"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4","Type":"ContainerStarted","Data":"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"}
Nov 03 20:20:29.318547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.318491    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.319145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.319110    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:29.331312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.331277    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.331617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.331582    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:29.435937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.435897    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.436226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.436194    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:29.842556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.842492    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.842747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.842583    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:29.910557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.909646    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.910557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:29.909723    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:30.048661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.048613    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.064758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.064696    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.159717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.159616    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.208858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.207629    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.304997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.304960    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.305282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.305250    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:30.330066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.330032    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.330765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.330722    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:30.330979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.330920    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.355545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.354724    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.393855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.393072    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.393855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.393479    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.427749 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.427637    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.428851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.428820    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.439271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.439239    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.439492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.439465    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:30.606812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.605506    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.606812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.606639    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.650837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.650771    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.904581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.896955    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.904581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.897027    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:30.921629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.907643    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.921629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.908043    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.921629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:30.908085    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:31.131694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.131645    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.209665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.157607    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:20:31.209665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.160356    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.209665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.160688    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:31.209665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.161049    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.372809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.372757    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.397235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.397203    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.522485    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.522578    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.528271    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.528334    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.531136    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.557556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.536763    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:31.610558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.605695    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.653901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.653842    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:31.679559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.654594    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:31.730557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.697167    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:20:31.730557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.607094    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.730557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.699328    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.730557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.716399    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:20:31.730557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.716720    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:31.941086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.940981    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:31.980631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.980590    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:31.988674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.959719    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.988876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.954733    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:31.989033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:31.989007    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:32.139664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.139621    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.154290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.154246    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.175452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.175410    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.204769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.204672    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.213671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.213635    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.380805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.380757    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.445555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.440654    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.445555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.440717    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.473722    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.473790    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.474434    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.475207    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.475249    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:32.493627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.492718    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.516681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.516635    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.546306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.546265    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.633299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.633248    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.741724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.741662    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.767618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.742717    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.869557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.863863    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.889507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.889462    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.889833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.889783    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:32.907284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.907254    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.907580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.907551    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:32.928451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:32.926658    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.369702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.369639    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.391614    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.392196    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.393326    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.393365    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.393975    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.406323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.394010    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:33.407117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.406953    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.433093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.407270    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.433093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.407601    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:33.433093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.430616    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.433093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.431704    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.433093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.432048    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.444917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.444009    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.444917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.444065    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:33.449413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.448613    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.507722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.505646    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.507722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.506073    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:20:33.507722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.506744    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" podUID="2cb0b7a8-707d-4413-b244-3e2f313b0e48" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.545707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.545628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.546400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.546380    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:33.569119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.569076    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.569767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.569746    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:33.685754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.685641    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.737676 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.737629    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.805558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.788067    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.805558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:33.788517    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.105703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.025773    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.105703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.094825    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:34.105703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.081266    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.105703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.081106    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.105703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.104092    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:34.170568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.152094    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:34.393788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.393674    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.450556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.443650    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.450556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.443716    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:34.450556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.443999    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.450556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.444036    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:34.470279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.467962    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.504861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.498756    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.504861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.498838    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:34.542562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.525771    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.601556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.548629    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.601556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.549024    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.601556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.549361    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:20:34.601556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.558632    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.662619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.604237    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:34.693743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.693699    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.770558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.748289    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.770558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.761068    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.784812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.776905    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.788746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.788701    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:34.831790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.831735    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.859559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.833154    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:34.859559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.846832    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.897586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.860095    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.897586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.864972    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.897586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.865038    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:34.920108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.901832    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.920403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:34.920363    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:35.118105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.118053    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:35.297171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.297135    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.301691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.301646    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:35.335169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.335133    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.335403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.335377    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:35.394723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.394681    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.441431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.441393    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.441721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.441693    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:35.475488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.475447    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.556614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.546662    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.556614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.548850    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.556614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.549263    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.631849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.631808    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:20:35.651154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.650061    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:35.729104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.729058    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.765555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.753177    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.775227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.775180    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.799109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.799065    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:20:35.901616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.889169    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.901616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.891359    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.919570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.917117    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.943899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.932041    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.943899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.932105    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:35.943899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.932842    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.943899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:35.932883    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:36.170808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.170699    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.329922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.329884    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.355668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.355624    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:36.355902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.349191    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 03 20:20:36.356038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.331685    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.356142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.356120    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:36.371755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.371691    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.372127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.372099    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Nov 03 20:20:36.413557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.410394    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:20:36.415156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.414364    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.416520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.416444    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.435515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.435324    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.435515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.435386    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 20:20:36.497467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.482632    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.565555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.560805    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.565555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.562339    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.571685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.571628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.680999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.680950    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.682921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.682894    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:36.772059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.772017    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.855559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.837294    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.855559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.837442    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.855559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.837778    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.855559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.851045    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.855559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.851097    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 20:20:36.895684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.895633    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.897222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.897189    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.953556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.915704    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.953556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.915767    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:36.953556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:36.943650    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.183755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.183652    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.306557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.305574    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.306557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.305638    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:37.378557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.366864    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.378557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.366933    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:37.426556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.425629    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:37.482615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.473746    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.506554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.486659    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.572772 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.572128    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.584556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.583642    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.707575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.696002    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.720563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.711115    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:20:37.759447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.726006    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:20:37.818821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.818790    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:20:37.851553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.840802    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.851553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.841205    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.899517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.898957    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.899990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.899964    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.900358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.900321    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 20:20:37.909460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.909432    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:37.919966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.919941    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:37.944217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.943914    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.961295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.961264    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:20:37.974963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.974867    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.985774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:37.985729    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.193244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.191937    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.217855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.217824    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:38.218500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.218477    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:38.218733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.218719    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:20:38.218877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.218867    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:38.218998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.218988    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:20:38.219114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.219105    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:20:38.219245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.219236    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:38.225027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.225001    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:38.241157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.239817    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:20:38.245397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.245380    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:38.254073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.254050    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:38.254382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.254359    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:38.295100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.295071    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:38.296324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.296296    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.296515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.296492    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 20:20:38.350398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.350360    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.352238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:38.352203    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:20:39.019953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.019918    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:39.021122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.021098    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:39.023330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.023303    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:20:39.301508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.301433    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:39.301973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.301956    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:39.302857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.302837    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:20:39.774052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.774017    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:39.775699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.775666    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:39.776902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:39.776878    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:20:41.970771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:41.970727    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.179556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.146559    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.181033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.180985    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.205665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.205618    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.261678 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.261618    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.262711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.262681    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.372561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.371665    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.447559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.446649    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.566622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.566327    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.566804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.566721    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.692557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.691780    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Nov 03 20:20:42.782557 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:20:42.768428    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:20:42.960690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.783807    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.990766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.888620    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.991693 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.888739    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.992135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.942652    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.002561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:42.971731    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.003565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.003517    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:43.181694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.181633    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.208694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.208661    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:43.223629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.210000    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.280556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.232271    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:43.302560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.283561    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.302560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.283939    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.302560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.284563    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:43.302560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.284623    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:20:43.383589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.376747    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.383589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.378272    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:43.423234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.423190    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.447848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.447805    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.448466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.448444    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:20:43.640557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.636255    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.729557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.728755    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.731114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.731074    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.731551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.731513    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:20:43.731655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.731600    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:20:43.814559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.757645    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.943933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.900193    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:43.943933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.900291    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:43.991638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.991581    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.018694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:43.999701    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.022396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.000176    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.040361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.015991    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.237670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.237613    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.292556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.291825    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.382885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.369400    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:44.382885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.369472    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:20:44.382885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.369509    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:20:44.382885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.369575    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:44.455552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.453839    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.648633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.643034    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.657624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.649546    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:20:44.813557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.803675    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:20:44.813557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.729030    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.818197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.790154    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.938861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.921624    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.938861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.922372    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:44.938861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.925089    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.938861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.925368    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.938861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.931612    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.982870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:44.963317    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.120635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.071853    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.120635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.108832    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.160557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.158252    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.282624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.268897    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.295571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.292875    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.463816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.463788    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:20:45.466591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.466558    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.592407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.592322    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:20:45.663554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.662938    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.814988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.814943    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.835732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.835683    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.960602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.958315    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.960602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.958668    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.960602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.958884    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.993241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:45.993177    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.122934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.122898    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123022    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123058    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123092    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123126    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123154    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123183    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123211    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123239    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:20:46.123378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.123267    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:20:46.132314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.132285    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:20:46.155391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.155359    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:20:46.186195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.186167    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:20:46.186891 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.186865    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:20:46.194564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.193767    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:20:46.194677 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:46.194648    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:20:49.460619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.460587    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"]
Nov 03 20:20:49.463583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.463567    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.480889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.480864    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"]
Nov 03 20:20:49.487394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.487371    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:49.487577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.487388    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:49.487657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.487408    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:49.487657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.487494    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:49.551280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.551239    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.551469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.551446    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfc5\" (UniqueName: \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.551652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.551618    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.653581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.653523    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfc5\" (UniqueName: \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.653804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.653737    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.654120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.654091    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.654603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.654573    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.656574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.656519    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.673079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.673054    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfc5\" (UniqueName: \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\") pod \"maas-api-revoke-sub-6247b024-hxhlc\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:49.778794 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:49.778764    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:50.094021 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:50.093973    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cfe63f_7002_493f_8c6a_a613b20d46fe.slice/crio-9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353 WatchSource:0}: Error finding container 9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353: Status 404 returned error can't find the container with id 9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353
Nov 03 20:20:50.095952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.095922    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"]
Nov 03 20:20:50.544251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.544213    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc" event={"ID":"34cfe63f-7002-493f-8c6a-a613b20d46fe","Type":"ContainerStarted","Data":"2c7cde5a36b7287a1732ba28aa869c2f89ecf4791da9269f2df6407b0ab3892e"}
Nov 03 20:20:50.544726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.544268    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc" event={"ID":"34cfe63f-7002-493f-8c6a-a613b20d46fe","Type":"ContainerStarted","Data":"9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353"}
Nov 03 20:20:50.596015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.595960    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc" podStartSLOduration=1.59594514 podStartE2EDuration="1.59594514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:50.582966626 +0000 UTC m=+992.791982621" watchObservedRunningTime="2026-11-03 20:20:50.59594514 +0000 UTC m=+992.804961135"
Nov 03 20:20:50.604619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.604594    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"]
Nov 03 20:20:50.607851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.607829    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.639565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.622928    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"]
Nov 03 20:20:50.670316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.670290    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxjs\" (UniqueName: \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.670497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.670430    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.670675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.670654    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.771761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.771726    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxjs\" (UniqueName: \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.771946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.771811    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.771946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.771928    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.772638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.772618    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.774517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.774488    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.793817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.793789    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxjs\" (UniqueName: \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\") pod \"maas-api-revoke-sub-6e0cf92d-zmjp5\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:50.950006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:50.949970    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:51.255260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.255228    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"]
Nov 03 20:20:51.255375 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:51.255275    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e2043_f1b3_4135_87ce_be4e86a01cf6.slice/crio-472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b WatchSource:0}: Error finding container 472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b: Status 404 returned error can't find the container with id 472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b
Nov 03 20:20:51.552549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.552492    5424 generic.go:356] "Generic (PLEG): container finished" podID="34cfe63f-7002-493f-8c6a-a613b20d46fe" containerID="2c7cde5a36b7287a1732ba28aa869c2f89ecf4791da9269f2df6407b0ab3892e" exitCode=0
Nov 03 20:20:51.552908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.552559    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc" event={"ID":"34cfe63f-7002-493f-8c6a-a613b20d46fe","Type":"ContainerDied","Data":"2c7cde5a36b7287a1732ba28aa869c2f89ecf4791da9269f2df6407b0ab3892e"}
Nov 03 20:20:51.554494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.554471    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5" event={"ID":"146e2043-f1b3-4135-87ce-be4e86a01cf6","Type":"ContainerStarted","Data":"2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad"}
Nov 03 20:20:51.554602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.554514    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5" event={"ID":"146e2043-f1b3-4135-87ce-be4e86a01cf6","Type":"ContainerStarted","Data":"472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b"}
Nov 03 20:20:51.629384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:51.629334    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5" podStartSLOduration=1.629320421 podStartE2EDuration="1.629320421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:51.626407296 +0000 UTC m=+993.835423292" watchObservedRunningTime="2026-11-03 20:20:51.629320421 +0000 UTC m=+993.838336467"
Nov 03 20:20:52.171226 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:20:52.171193    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e2043_f1b3_4135_87ce_be4e86a01cf6.slice/crio-conmon-2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e2043_f1b3_4135_87ce_be4e86a01cf6.slice/crio-2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:20:52.564105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.564070    5424 generic.go:356] "Generic (PLEG): container finished" podID="146e2043-f1b3-4135-87ce-be4e86a01cf6" containerID="2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad" exitCode=0
Nov 03 20:20:52.564632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.564122    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5" event={"ID":"146e2043-f1b3-4135-87ce-be4e86a01cf6","Type":"ContainerDied","Data":"2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad"}
Nov 03 20:20:52.721514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.721490    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:52.805233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.805193    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\") pod \"34cfe63f-7002-493f-8c6a-a613b20d46fe\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") "
Nov 03 20:20:52.805401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.805300    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\" (UniqueName: \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\") pod \"34cfe63f-7002-493f-8c6a-a613b20d46fe\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") "
Nov 03 20:20:52.805564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.805520    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\") pod \"34cfe63f-7002-493f-8c6a-a613b20d46fe\" (UID: \"34cfe63f-7002-493f-8c6a-a613b20d46fe\") "
Nov 03 20:20:52.805564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.805552    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca" pod "34cfe63f-7002-493f-8c6a-a613b20d46fe" (UID: "34cfe63f-7002-493f-8c6a-a613b20d46fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:52.806143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.806123    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:52.807697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.807668    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca" pod "34cfe63f-7002-493f-8c6a-a613b20d46fe" (UID: "34cfe63f-7002-493f-8c6a-a613b20d46fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:52.807780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.807741    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5" pod "34cfe63f-7002-493f-8c6a-a613b20d46fe" (UID: "34cfe63f-7002-493f-8c6a-a613b20d46fe"). InnerVolumeSpecName "kube-api-access-6tfc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:52.907424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.907383    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cfe63f-7002-493f-8c6a-a613b20d46fe-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:52.907424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:52.907416    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfc5\" (UniqueName: \"kubernetes.io/projected/34cfe63f-7002-493f-8c6a-a613b20d46fe-kube-api-access-6tfc5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.583780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.583751    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"
Nov 03 20:20:53.584172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.583775    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc" event={"ID":"34cfe63f-7002-493f-8c6a-a613b20d46fe","Type":"ContainerDied","Data":"9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353"}
Nov 03 20:20:53.584172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.583845    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eee2be4a321d8e04faed1ad14f399e912dc2d38f917199872515b0474be1353"
Nov 03 20:20:53.750326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.750303    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:53.821080 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.821048    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\" (UniqueName: \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\") pod \"146e2043-f1b3-4135-87ce-be4e86a01cf6\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") "
Nov 03 20:20:53.821261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.821245    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\") pod \"146e2043-f1b3-4135-87ce-be4e86a01cf6\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") "
Nov 03 20:20:53.821501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.821482    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\") pod \"146e2043-f1b3-4135-87ce-be4e86a01cf6\" (UID: \"146e2043-f1b3-4135-87ce-be4e86a01cf6\") "
Nov 03 20:20:53.821947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.821913    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca" pod "146e2043-f1b3-4135-87ce-be4e86a01cf6" (UID: "146e2043-f1b3-4135-87ce-be4e86a01cf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:53.822354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.822329    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.823485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.823461    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs" pod "146e2043-f1b3-4135-87ce-be4e86a01cf6" (UID: "146e2043-f1b3-4135-87ce-be4e86a01cf6"). InnerVolumeSpecName "kube-api-access-rtxjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:53.824101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.824068    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca" pod "146e2043-f1b3-4135-87ce-be4e86a01cf6" (UID: "146e2043-f1b3-4135-87ce-be4e86a01cf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:53.924259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.924181    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146e2043-f1b3-4135-87ce-be4e86a01cf6-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.924259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:53.924207    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxjs\" (UniqueName: \"kubernetes.io/projected/146e2043-f1b3-4135-87ce-be4e86a01cf6-kube-api-access-rtxjs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:54.593850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:54.593810    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5" event={"ID":"146e2043-f1b3-4135-87ce-be4e86a01cf6","Type":"ContainerDied","Data":"472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b"}
Nov 03 20:20:54.593850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:54.593835    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"
Nov 03 20:20:54.594371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:54.593915    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="472b88c575f1d3fca54f569331311325da8f9cf31da5729fa02c2e04ea724e0b"
Nov 03 20:20:56.241670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.241638    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"]
Nov 03 20:20:56.242295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242276    5424 cpu_manager.go:414] "containerMap: removing container" podUID="146e2043-f1b3-4135-87ce-be4e86a01cf6" containerName="revoke-keys"
Nov 03 20:20:56.242372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242298    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146e2043-f1b3-4135-87ce-be4e86a01cf6" containerName="revoke-keys"
Nov 03 20:20:56.242372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242343    5424 cpu_manager.go:414] "containerMap: removing container" podUID="34cfe63f-7002-493f-8c6a-a613b20d46fe" containerName="revoke-keys"
Nov 03 20:20:56.242372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242349    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34cfe63f-7002-493f-8c6a-a613b20d46fe" containerName="revoke-keys"
Nov 03 20:20:56.242488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242478    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="146e2043-f1b3-4135-87ce-be4e86a01cf6" containerName="revoke-keys"
Nov 03 20:20:56.242523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.242493    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="34cfe63f-7002-493f-8c6a-a613b20d46fe" containerName="revoke-keys"
Nov 03 20:20:56.245025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.245011    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.261335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.261311    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"]
Nov 03 20:20:56.268460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.268409    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:56.268637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.268490    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:56.268637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.268507    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:56.271631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.271613    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:56.359971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.359939    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.360138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.359986    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.360257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.360236    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.461521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.461490    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.461764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.461572    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.461764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.461749    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.462154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.462129    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.464188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.464168    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.481320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.481290    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\") pod \"maas-api-revoke-sub-60ed8e6f-rnwp9\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.558261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.558175    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:56.865426 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:20:56.865393    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54b8845_c35f_43fe_893e_486bb7b28ad2.slice/crio-9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763 WatchSource:0}: Error finding container 9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763: Status 404 returned error can't find the container with id 9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763
Nov 03 20:20:56.866348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:56.866321    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"]
Nov 03 20:20:57.625040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:57.625000    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9" event={"ID":"f54b8845-c35f-43fe-893e-486bb7b28ad2","Type":"ContainerStarted","Data":"11f4f1466535fdc7eb5b37ce0b609f46c1f4ee16bc408658b019d0f31ddd927c"}
Nov 03 20:20:57.625040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:57.625057    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9" event={"ID":"f54b8845-c35f-43fe-893e-486bb7b28ad2","Type":"ContainerStarted","Data":"9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763"}
Nov 03 20:20:57.664446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:57.664400    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9" podStartSLOduration=1.664385546 podStartE2EDuration="1.664385546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:57.662200021 +0000 UTC m=+999.871216019" watchObservedRunningTime="2026-11-03 20:20:57.664385546 +0000 UTC m=+999.873401541"
Nov 03 20:20:58.633849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:58.633812    5424 generic.go:356] "Generic (PLEG): container finished" podID="f54b8845-c35f-43fe-893e-486bb7b28ad2" containerID="11f4f1466535fdc7eb5b37ce0b609f46c1f4ee16bc408658b019d0f31ddd927c" exitCode=0
Nov 03 20:20:58.634233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:58.633902    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9" event={"ID":"f54b8845-c35f-43fe-893e-486bb7b28ad2","Type":"ContainerDied","Data":"11f4f1466535fdc7eb5b37ce0b609f46c1f4ee16bc408658b019d0f31ddd927c"}
Nov 03 20:20:59.803651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.803628    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:20:59.910020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.909939    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\") pod \"f54b8845-c35f-43fe-893e-486bb7b28ad2\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") "
Nov 03 20:20:59.910294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.910275    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\") pod \"f54b8845-c35f-43fe-893e-486bb7b28ad2\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") "
Nov 03 20:20:59.910368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.910322    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\") pod \"f54b8845-c35f-43fe-893e-486bb7b28ad2\" (UID: \"f54b8845-c35f-43fe-893e-486bb7b28ad2\") "
Nov 03 20:20:59.910725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.910701    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca" pod "f54b8845-c35f-43fe-893e-486bb7b28ad2" (UID: "f54b8845-c35f-43fe-893e-486bb7b28ad2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:59.910941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.910926    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:20:59.912698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.912669    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca" pod "f54b8845-c35f-43fe-893e-486bb7b28ad2" (UID: "f54b8845-c35f-43fe-893e-486bb7b28ad2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:59.912791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:20:59.912695    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9" pod "f54b8845-c35f-43fe-893e-486bb7b28ad2" (UID: "f54b8845-c35f-43fe-893e-486bb7b28ad2"). InnerVolumeSpecName "kube-api-access-s5jc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:00.013058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.013024    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54b8845-c35f-43fe-893e-486bb7b28ad2-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:00.013058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.013055    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/f54b8845-c35f-43fe-893e-486bb7b28ad2-kube-api-access-s5jc9\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:00.280447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.280419    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"]
Nov 03 20:21:00.281009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.280996    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f54b8845-c35f-43fe-893e-486bb7b28ad2" containerName="revoke-keys"
Nov 03 20:21:00.281060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.281010    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54b8845-c35f-43fe-893e-486bb7b28ad2" containerName="revoke-keys"
Nov 03 20:21:00.281110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.281099    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54b8845-c35f-43fe-893e-486bb7b28ad2" containerName="revoke-keys"
Nov 03 20:21:00.283835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.283818    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.301578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.301553    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"]
Nov 03 20:21:00.418209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.418177    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.418390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.418336    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjwt8\" (UniqueName: \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.418485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.418462    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.519376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.519348    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjwt8\" (UniqueName: \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.519573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.519429    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.519621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.519571    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.520061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.520042    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.522055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.522031    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.538204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.538144    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjwt8\" (UniqueName: \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\") pod \"maas-api-revoke-sub-21caae8b-s2kvx\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.598620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.598597    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:00.657910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.656609    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"
Nov 03 20:21:00.657910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.656679    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9" event={"ID":"f54b8845-c35f-43fe-893e-486bb7b28ad2","Type":"ContainerDied","Data":"9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763"}
Nov 03 20:21:00.657910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.656721    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9074091344e47f887b4aa69d651b8fd5c465d205c559d6f14ed72b5ab7303763"
Nov 03 20:21:00.900210 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:00.900182    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb016d30e_2d15_4de3_a6dd_180c057a717a.slice/crio-0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a WatchSource:0}: Error finding container 0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a: Status 404 returned error can't find the container with id 0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a
Nov 03 20:21:00.905303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:00.905277    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"]
Nov 03 20:21:01.682842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:01.682786    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx" event={"ID":"b016d30e-2d15-4de3-a6dd-180c057a717a","Type":"ContainerStarted","Data":"ef38861cd07b6a7024610bf53fd53d1576eb917c2c61b7cf6dac2a279a0a9061"}
Nov 03 20:21:01.683746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:01.683710    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx" event={"ID":"b016d30e-2d15-4de3-a6dd-180c057a717a","Type":"ContainerStarted","Data":"0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a"}
Nov 03 20:21:01.727570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:01.727501    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx" podStartSLOduration=1.727487756 podStartE2EDuration="1.727487756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:01.723340694 +0000 UTC m=+1003.932356691" watchObservedRunningTime="2026-11-03 20:21:01.727487756 +0000 UTC m=+1003.936503808"
Nov 03 20:21:02.693246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:02.693210    5424 generic.go:356] "Generic (PLEG): container finished" podID="b016d30e-2d15-4de3-a6dd-180c057a717a" containerID="ef38861cd07b6a7024610bf53fd53d1576eb917c2c61b7cf6dac2a279a0a9061" exitCode=0
Nov 03 20:21:02.693731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:02.693285    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx" event={"ID":"b016d30e-2d15-4de3-a6dd-180c057a717a","Type":"ContainerDied","Data":"ef38861cd07b6a7024610bf53fd53d1576eb917c2c61b7cf6dac2a279a0a9061"}
Nov 03 20:21:03.864138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.864113    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:03.977777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.977682    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\") pod \"b016d30e-2d15-4de3-a6dd-180c057a717a\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") "
Nov 03 20:21:03.977937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.977793    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\" (UniqueName: \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\") pod \"b016d30e-2d15-4de3-a6dd-180c057a717a\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") "
Nov 03 20:21:03.977937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.977883    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\") pod \"b016d30e-2d15-4de3-a6dd-180c057a717a\" (UID: \"b016d30e-2d15-4de3-a6dd-180c057a717a\") "
Nov 03 20:21:03.978171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.978133    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca" pod "b016d30e-2d15-4de3-a6dd-180c057a717a" (UID: "b016d30e-2d15-4de3-a6dd-180c057a717a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:03.978917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.978853    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:03.980562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.980496    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca" pod "b016d30e-2d15-4de3-a6dd-180c057a717a" (UID: "b016d30e-2d15-4de3-a6dd-180c057a717a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:03.980654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:03.980597    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8" pod "b016d30e-2d15-4de3-a6dd-180c057a717a" (UID: "b016d30e-2d15-4de3-a6dd-180c057a717a"). InnerVolumeSpecName "kube-api-access-rjwt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:04.080944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.080907    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjwt8\" (UniqueName: \"kubernetes.io/projected/b016d30e-2d15-4de3-a6dd-180c057a717a-kube-api-access-rjwt8\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:04.080944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.080936    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b016d30e-2d15-4de3-a6dd-180c057a717a-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:04.107270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.107229    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"]
Nov 03 20:21:04.107801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.107783    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b016d30e-2d15-4de3-a6dd-180c057a717a" containerName="revoke-keys"
Nov 03 20:21:04.107801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.107798    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b016d30e-2d15-4de3-a6dd-180c057a717a" containerName="revoke-keys"
Nov 03 20:21:04.107971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.107942    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b016d30e-2d15-4de3-a6dd-180c057a717a" containerName="revoke-keys"
Nov 03 20:21:04.111482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.111461    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.129144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.129121    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"]
Nov 03 20:21:04.284611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.284579    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.284808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.284790    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.284854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.284843    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2pf\" (UniqueName: \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.385922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.385889    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.386073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.386060    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.386132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.386105    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2pf\" (UniqueName: \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.386498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.386478    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.388709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.388688    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.405086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.405060    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2pf\" (UniqueName: \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\") pod \"maas-api-revoke-sub-dc4307b6-k7tbf\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.427935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.427910    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:04.710168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.710097    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"
Nov 03 20:21:04.710325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.710092    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx" event={"ID":"b016d30e-2d15-4de3-a6dd-180c057a717a","Type":"ContainerDied","Data":"0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a"}
Nov 03 20:21:04.710325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.710214    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0475d2c926c803962d62c8d762101812577e3a609336a911a20be24c8364cb7a"
Nov 03 20:21:04.737031 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:04.737009    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc45015a_8ca6_488c_9ef1_ea4c0af7e750.slice/crio-a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191 WatchSource:0}: Error finding container a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191: Status 404 returned error can't find the container with id a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191
Nov 03 20:21:04.738963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:04.738936    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"]
Nov 03 20:21:05.719738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:05.719702    5424 generic.go:356] "Generic (PLEG): container finished" podID="dc45015a-8ca6-488c-9ef1-ea4c0af7e750" containerID="26feefa7b1688ecbf42e1cd94869a0dbdec4d8ba21350d0a6ca65a834f1c13bc" exitCode=0
Nov 03 20:21:05.720166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:05.719756    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf" event={"ID":"dc45015a-8ca6-488c-9ef1-ea4c0af7e750","Type":"ContainerDied","Data":"26feefa7b1688ecbf42e1cd94869a0dbdec4d8ba21350d0a6ca65a834f1c13bc"}
Nov 03 20:21:05.720166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:05.719837    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf" event={"ID":"dc45015a-8ca6-488c-9ef1-ea4c0af7e750","Type":"ContainerStarted","Data":"a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191"}
Nov 03 20:21:06.885877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:06.885855    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:07.026189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.026151    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\") pod \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") "
Nov 03 20:21:07.026380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.026296    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\" (UniqueName: \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\") pod \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") "
Nov 03 20:21:07.026484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.026465    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\") pod \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\" (UID: \"dc45015a-8ca6-488c-9ef1-ea4c0af7e750\") "
Nov 03 20:21:07.026603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.026512    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca" pod "dc45015a-8ca6-488c-9ef1-ea4c0af7e750" (UID: "dc45015a-8ca6-488c-9ef1-ea4c0af7e750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:07.027122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.027091    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:07.028611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.028588    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca" pod "dc45015a-8ca6-488c-9ef1-ea4c0af7e750" (UID: "dc45015a-8ca6-488c-9ef1-ea4c0af7e750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:07.028674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.028637    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf" pod "dc45015a-8ca6-488c-9ef1-ea4c0af7e750" (UID: "dc45015a-8ca6-488c-9ef1-ea4c0af7e750"). InnerVolumeSpecName "kube-api-access-zt2pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:07.128216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.128178    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2pf\" (UniqueName: \"kubernetes.io/projected/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-kube-api-access-zt2pf\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:07.128216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.128217    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc45015a-8ca6-488c-9ef1-ea4c0af7e750-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:07.573122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.573091    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"]
Nov 03 20:21:07.573663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.573648    5424 cpu_manager.go:414] "containerMap: removing container" podUID="dc45015a-8ca6-488c-9ef1-ea4c0af7e750" containerName="revoke-keys"
Nov 03 20:21:07.573717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.573666    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc45015a-8ca6-488c-9ef1-ea4c0af7e750" containerName="revoke-keys"
Nov 03 20:21:07.573820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.573810    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc45015a-8ca6-488c-9ef1-ea4c0af7e750" containerName="revoke-keys"
Nov 03 20:21:07.577420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.577402    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.593894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.593866    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"]
Nov 03 20:21:07.737147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737110    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzrk\" (UniqueName: \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.737339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737207    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"
Nov 03 20:21:07.737339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737217    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf" event={"ID":"dc45015a-8ca6-488c-9ef1-ea4c0af7e750","Type":"ContainerDied","Data":"a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191"}
Nov 03 20:21:07.737339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737261    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4ea84435e3ca3db04fdd0912f6270141de7ac699180c7706f4c2ba853957191"
Nov 03 20:21:07.737339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737323    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.737549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.737341    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.838284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.838212    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzrk\" (UniqueName: \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.838424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.838284    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.838424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.838304    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.839008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.838984    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.841048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.841021    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.856922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.856899    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzrk\" (UniqueName: \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\") pod \"maas-api-revoke-sub-e60e6ebf-z7wcs\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:07.892320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:07.892296    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:08.213731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:08.211394    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"]
Nov 03 20:21:08.746895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:08.746860    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs" event={"ID":"b489a858-3f0a-4924-8602-5c1f2fd4f5f8","Type":"ContainerStarted","Data":"4dc3746237da5b0efa60d3773965c275c89711f2797e988656da65b4b982f9c5"}
Nov 03 20:21:08.747108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:08.746923    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs" event={"ID":"b489a858-3f0a-4924-8602-5c1f2fd4f5f8","Type":"ContainerStarted","Data":"6b555e4e1a5cf445d4d8a1bb8105f333640d5707e08de28530e929465bff9bc9"}
Nov 03 20:21:08.788497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:08.788395    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs" podStartSLOduration=1.7883817290000001 podStartE2EDuration="1.788381729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:08.783444746 +0000 UTC m=+1010.992460824" watchObservedRunningTime="2026-11-03 20:21:08.788381729 +0000 UTC m=+1010.997397725"
Nov 03 20:21:09.756415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:09.756328    5424 generic.go:356] "Generic (PLEG): container finished" podID="b489a858-3f0a-4924-8602-5c1f2fd4f5f8" containerID="4dc3746237da5b0efa60d3773965c275c89711f2797e988656da65b4b982f9c5" exitCode=0
Nov 03 20:21:09.756415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:09.756386    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs" event={"ID":"b489a858-3f0a-4924-8602-5c1f2fd4f5f8","Type":"ContainerDied","Data":"4dc3746237da5b0efa60d3773965c275c89711f2797e988656da65b4b982f9c5"}
Nov 03 20:21:10.937001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.936970    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:10.986479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.986448    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\") pod \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") "
Nov 03 20:21:10.986689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.986668    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\") pod \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") "
Nov 03 20:21:10.986862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.986845    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\" (UniqueName: \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\") pod \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\" (UID: \"b489a858-3f0a-4924-8602-5c1f2fd4f5f8\") "
Nov 03 20:21:10.986933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.986894    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca" pod "b489a858-3f0a-4924-8602-5c1f2fd4f5f8" (UID: "b489a858-3f0a-4924-8602-5c1f2fd4f5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:10.988057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.987723    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:10.989216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.989191    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk" pod "b489a858-3f0a-4924-8602-5c1f2fd4f5f8" (UID: "b489a858-3f0a-4924-8602-5c1f2fd4f5f8"). InnerVolumeSpecName "kube-api-access-ngzrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:10.989446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:10.989424    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca" pod "b489a858-3f0a-4924-8602-5c1f2fd4f5f8" (UID: "b489a858-3f0a-4924-8602-5c1f2fd4f5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:11.089199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.089129    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:11.089199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.089154    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzrk\" (UniqueName: \"kubernetes.io/projected/b489a858-3f0a-4924-8602-5c1f2fd4f5f8-kube-api-access-ngzrk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:11.109733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.109701    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"]
Nov 03 20:21:11.110516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.110500    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b489a858-3f0a-4924-8602-5c1f2fd4f5f8" containerName="revoke-keys"
Nov 03 20:21:11.110614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.110517    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b489a858-3f0a-4924-8602-5c1f2fd4f5f8" containerName="revoke-keys"
Nov 03 20:21:11.110723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.110711    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b489a858-3f0a-4924-8602-5c1f2fd4f5f8" containerName="revoke-keys"
Nov 03 20:21:11.114927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.114903    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.131549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.131498    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"]
Nov 03 20:21:11.190836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.190804    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tzhp\" (UniqueName: \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.191013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.190852    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.191062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.191011    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.295550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.293276    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.295550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.293626    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tzhp\" (UniqueName: \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.295550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.293684    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.295550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.294699    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.299661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.299272    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.316727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.316687    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tzhp\" (UniqueName: \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\") pod \"maas-api-revoke-sub-fd1aa0db-sg9cs\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.433785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.433703    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:11.581971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.581938    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"]
Nov 03 20:21:11.587409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.587390    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.602782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.602759    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"]
Nov 03 20:21:11.700842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.700769    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.701000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.700849    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.701000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.700923    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqk5\" (UniqueName: \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.756279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.756242    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"]
Nov 03 20:21:11.803570 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:11.793112    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537 WatchSource:0}: Error finding container c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537: Status 404 returned error can't find the container with id c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537
Nov 03 20:21:11.863681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.861841    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs" event={"ID":"b489a858-3f0a-4924-8602-5c1f2fd4f5f8","Type":"ContainerDied","Data":"6b555e4e1a5cf445d4d8a1bb8105f333640d5707e08de28530e929465bff9bc9"}
Nov 03 20:21:11.863681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.861908    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b555e4e1a5cf445d4d8a1bb8105f333640d5707e08de28530e929465bff9bc9"
Nov 03 20:21:11.863681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.861985    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"
Nov 03 20:21:11.936610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.933677    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.936610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.933841    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.936610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.933890    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqk5\" (UniqueName: \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.938008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.937979    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.942968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.942888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:11.955915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:11.955873    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqk5\" (UniqueName: \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\") pod \"maas-api-revoke-sub-25ae90ca-cnkk6\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:12.212638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:12.208956    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:12.398586 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:12.396778    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:21:13.294815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.271920    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs" event={"ID":"e13b617e-38de-40e6-9f35-87629d33a97c","Type":"ContainerStarted","Data":"a62e12b8e41d82bb4ac7d4e518f3c65ff526d4897a3b85a13aaac6ffdc523f90"}
Nov 03 20:21:13.294815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.271990    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs" event={"ID":"e13b617e-38de-40e6-9f35-87629d33a97c","Type":"ContainerStarted","Data":"c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537"}
Nov 03 20:21:13.316716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.316671    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"]
Nov 03 20:21:13.354371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.354307    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs" podStartSLOduration=2.354283513 podStartE2EDuration="2.354283513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:13.335000665 +0000 UTC m=+1015.544016663" watchObservedRunningTime="2026-11-03 20:21:13.354283513 +0000 UTC m=+1015.563299512"
Nov 03 20:21:13.458590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.445611    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.765922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.764639    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.792056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.791626    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.809511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.809116    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:13.858865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.857623    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.859131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.859101    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.940557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.938982    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.960975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:13.959570    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.050556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.049669    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.162744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.162642    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.163360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.163324    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.165548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.165161    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.214695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.209006    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.214695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.209367    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.362128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.362083    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6" event={"ID":"270d6e52-0b79-4b9c-806a-64f2ca779fa0","Type":"ContainerStarted","Data":"35ed5969a6cfcd8eb3b5ecff58b98207dcd4904e130e85d1a65a7256d64c9cce"}
Nov 03 20:21:14.368713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.368641    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6" event={"ID":"270d6e52-0b79-4b9c-806a-64f2ca779fa0","Type":"ContainerStarted","Data":"890cea60d0cad41c1ec3fee73594d036315fa019114a20301d96c7a704a50897"}
Nov 03 20:21:14.375271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.375236    5424 generic.go:356] "Generic (PLEG): container finished" podID="e13b617e-38de-40e6-9f35-87629d33a97c" containerID="a62e12b8e41d82bb4ac7d4e518f3c65ff526d4897a3b85a13aaac6ffdc523f90" exitCode=0
Nov 03 20:21:14.375584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.375561    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs" event={"ID":"e13b617e-38de-40e6-9f35-87629d33a97c","Type":"ContainerDied","Data":"a62e12b8e41d82bb4ac7d4e518f3c65ff526d4897a3b85a13aaac6ffdc523f90"}
Nov 03 20:21:14.433595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.433442    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6" podStartSLOduration=3.433420362 podStartE2EDuration="3.433420362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:14.410953176 +0000 UTC m=+1016.619969176" watchObservedRunningTime="2026-11-03 20:21:14.433420362 +0000 UTC m=+1016.642436360"
Nov 03 20:21:14.480553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.472640    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.480553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.473144    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.799556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.796993    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.799556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.798331    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.799556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.799068    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:21:14.799556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.799129    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:21:14.818441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.818389    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.819317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.819283    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:14.948554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.929698    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.948554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:14.930455    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:21:15.029249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.029149    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.056909    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.062693    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.063388    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.116662    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.173114    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.173157    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.173194    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.173228    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.166824    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.166876    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.166898    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.174983    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.175028    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:21:15.175571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.175062    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:21:15.233858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.233718    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.246578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.225944    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.450646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.381398    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:21:15.450646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.445317    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:15.484817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.455133    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:15.577592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.530107    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.577592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.533602    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.577592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.571623    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:21:15.577592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.571687    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:21:15.921737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.921666    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.968603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:15.953087    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.061001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.047292    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.097170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.097129    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.177264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.176841    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.179148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.177553    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.179148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.177771    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.179148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.178246    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.179434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.179041    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.179948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.179778    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.180089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.179721    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.239596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.239310    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.418632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.411216    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.418632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.411695    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 03 20:21:16.418632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.412224    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:16.418632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.414659    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.512602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.499562    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.512602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.509795    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.589554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.575632    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.589554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.575957    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.883057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.882974    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:16.913561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.912039    5424 generic.go:356] "Generic (PLEG): container finished" podID="270d6e52-0b79-4b9c-806a-64f2ca779fa0" containerID="35ed5969a6cfcd8eb3b5ecff58b98207dcd4904e130e85d1a65a7256d64c9cce" exitCode=0
Nov 03 20:21:16.913561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.913060    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6" event={"ID":"270d6e52-0b79-4b9c-806a-64f2ca779fa0","Type":"ContainerDied","Data":"35ed5969a6cfcd8eb3b5ecff58b98207dcd4904e130e85d1a65a7256d64c9cce"}
Nov 03 20:21:16.923668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.922617    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.958721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:16.958677    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.069597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.055307    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.160392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.160282    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.184950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.184306    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.188716    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.217842    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.218104    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.218201    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.218315    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.543612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.230399    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:17.579119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.579075    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.599870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.599815    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.600442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.415861    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.638596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.275166    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.665555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.649288    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.666892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.666861    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:17.732558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.716206    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\") pod \"e13b617e-38de-40e6-9f35-87629d33a97c\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") "
Nov 03 20:21:17.758558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.732877    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\") pod \"e13b617e-38de-40e6-9f35-87629d33a97c\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") "
Nov 03 20:21:17.881680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.809026    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca" pod "e13b617e-38de-40e6-9f35-87629d33a97c" (UID: "e13b617e-38de-40e6-9f35-87629d33a97c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:17.909150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.908933    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca" pod "e13b617e-38de-40e6-9f35-87629d33a97c" (UID: "e13b617e-38de-40e6-9f35-87629d33a97c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:17.921554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.881589    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\" (UniqueName: \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\") pod \"e13b617e-38de-40e6-9f35-87629d33a97c\" (UID: \"e13b617e-38de-40e6-9f35-87629d33a97c\") "
Nov 03 20:21:18.176738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:17.926658    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.177073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.023657    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 20:21:18.177227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.023969    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.177414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.057670    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.188582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.188519    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.564706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.564639    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.566368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.566329    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.611482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.611447    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.612177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.572646    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.612964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.612694    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.613070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.572612    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.618619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.612644    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.632940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.632915    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:21:18.646333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.646300    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.697564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.685426    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13b617e-38de-40e6-9f35-87629d33a97c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.701358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.572667    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:18.721710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.719265    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp" pod "e13b617e-38de-40e6-9f35-87629d33a97c" (UID: "e13b617e-38de-40e6-9f35-87629d33a97c"). InnerVolumeSpecName "kube-api-access-7tzhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:18.734025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.733656    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.735887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.735852    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.738075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.738047    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.779848    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs" event={"ID":"e13b617e-38de-40e6-9f35-87629d33a97c","Type":"ContainerDied","Data":"c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537"}
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.779921    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.779984    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.780033    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.780064    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.780100    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.780136    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.786000    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.792196    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.793746    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:21:18.799334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.794267    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:21:18.801189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.800141    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tzhp\" (UniqueName: \"kubernetes.io/projected/e13b617e-38de-40e6-9f35-87629d33a97c-kube-api-access-7tzhp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.835605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.830427    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:18.900549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.864498    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 20:21:18.952086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:18.952057    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:21:19.006474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.006307    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:21:19.064726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.064700    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:19.068941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.068809    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:19.069084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.069051    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:21:19.072428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.072409    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:21:19.095732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.095661    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:21:19.157875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.157837    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:19.233565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.233494    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\") pod \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") "
Nov 03 20:21:19.233797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.233697    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\" (UniqueName: \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\") pod \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") "
Nov 03 20:21:19.233797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.233740    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\") pod \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\" (UID: \"270d6e52-0b79-4b9c-806a-64f2ca779fa0\") "
Nov 03 20:21:19.234960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.234933    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca" pod "270d6e52-0b79-4b9c-806a-64f2ca779fa0" (UID: "270d6e52-0b79-4b9c-806a-64f2ca779fa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:19.236894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.236867    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca" pod "270d6e52-0b79-4b9c-806a-64f2ca779fa0" (UID: "270d6e52-0b79-4b9c-806a-64f2ca779fa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:19.238345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.238320    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5" pod "270d6e52-0b79-4b9c-806a-64f2ca779fa0" (UID: "270d6e52-0b79-4b9c-806a-64f2ca779fa0"). InnerVolumeSpecName "kube-api-access-rrqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:19.336512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.336477    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:19.336680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.336517    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrqk5\" (UniqueName: \"kubernetes.io/projected/270d6e52-0b79-4b9c-806a-64f2ca779fa0-kube-api-access-rrqk5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:19.336680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.336572    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270d6e52-0b79-4b9c-806a-64f2ca779fa0-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:19.643845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.643815    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"
Nov 03 20:21:19.644214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.643850    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6" event={"ID":"270d6e52-0b79-4b9c-806a-64f2ca779fa0","Type":"ContainerDied","Data":"890cea60d0cad41c1ec3fee73594d036315fa019114a20301d96c7a704a50897"}
Nov 03 20:21:19.644214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:19.643943    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890cea60d0cad41c1ec3fee73594d036315fa019114a20301d96c7a704a50897"
Nov 03 20:21:20.540408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.540370    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"]
Nov 03 20:21:20.540946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.540932    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e13b617e-38de-40e6-9f35-87629d33a97c" containerName="revoke-keys"
Nov 03 20:21:20.540993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.540948    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13b617e-38de-40e6-9f35-87629d33a97c" containerName="revoke-keys"
Nov 03 20:21:20.540993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.540972    5424 cpu_manager.go:414] "containerMap: removing container" podUID="270d6e52-0b79-4b9c-806a-64f2ca779fa0" containerName="revoke-keys"
Nov 03 20:21:20.540993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.540978    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270d6e52-0b79-4b9c-806a-64f2ca779fa0" containerName="revoke-keys"
Nov 03 20:21:20.541108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.541098    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="270d6e52-0b79-4b9c-806a-64f2ca779fa0" containerName="revoke-keys"
Nov 03 20:21:20.541138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.541114    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13b617e-38de-40e6-9f35-87629d33a97c" containerName="revoke-keys"
Nov 03 20:21:20.544874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.544856    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.562177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.562149    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"]
Nov 03 20:21:20.567573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.567550    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:20.567703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.567632    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:21:20.571430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.571407    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:20.571571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.571490    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:21:20.656494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.656460    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.656898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.656714    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svwj\" (UniqueName: \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.656898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.656742    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.758143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.758112    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svwj\" (UniqueName: \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.758293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.758151    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.758293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.758264    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.758874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.758856    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.760855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.760832    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.777218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.777182    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svwj\" (UniqueName: \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\") pod \"maas-api-revoke-sub-2f47421c-hprfc\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:20.859461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:20.859378    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:21.166455 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:21.166424    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886aecdc_b29b_40de_ab51_e1ec518bdbf5.slice/crio-402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975 WatchSource:0}: Error finding container 402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975: Status 404 returned error can't find the container with id 402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975
Nov 03 20:21:21.167319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.167293    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"]
Nov 03 20:21:21.662767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.662731    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc" event={"ID":"886aecdc-b29b-40de-ab51-e1ec518bdbf5","Type":"ContainerStarted","Data":"be23ab477f846dd4fcd43dd2c67b5c061ff0f4e8ff3d3a176c3f81375458d862"}
Nov 03 20:21:21.663116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.662791    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc" event={"ID":"886aecdc-b29b-40de-ab51-e1ec518bdbf5","Type":"ContainerStarted","Data":"402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975"}
Nov 03 20:21:21.704817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.704719    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc" podStartSLOduration=1.704704744 podStartE2EDuration="1.704704744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:21.700676148 +0000 UTC m=+1023.909692145" watchObservedRunningTime="2026-11-03 20:21:21.704704744 +0000 UTC m=+1023.913720740"
Nov 03 20:21:21.975643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.975567    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"]
Nov 03 20:21:21.981397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:21.981370    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.002738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.002714    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"]
Nov 03 20:21:22.083507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.083464    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.083715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.083511    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xv27\" (UniqueName: \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.083835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.083806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.180236 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:22.180199    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:22.187487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.187455    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.187666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.187502    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xv27\" (UniqueName: \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.187706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.187666    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.188325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.188291    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.190521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.190493    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.210306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.210279    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xv27\" (UniqueName: \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\") pod \"maas-api-revoke-sub-ed8028c2-f49vj\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.298189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.298149    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:22.611837 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:22.611805    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c174cb2_8f31_40e9_b54c_c6e3c14daaca.slice/crio-ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc WatchSource:0}: Error finding container ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc: Status 404 returned error can't find the container with id ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc
Nov 03 20:21:22.613700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.613672    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"]
Nov 03 20:21:22.678951 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.678902    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj" event={"ID":"5c174cb2-8f31-40e9-b54c-c6e3c14daaca","Type":"ContainerStarted","Data":"ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc"}
Nov 03 20:21:22.680995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.680969    5424 generic.go:356] "Generic (PLEG): container finished" podID="886aecdc-b29b-40de-ab51-e1ec518bdbf5" containerID="be23ab477f846dd4fcd43dd2c67b5c061ff0f4e8ff3d3a176c3f81375458d862" exitCode=0
Nov 03 20:21:22.681133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:22.681055    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc" event={"ID":"886aecdc-b29b-40de-ab51-e1ec518bdbf5","Type":"ContainerDied","Data":"be23ab477f846dd4fcd43dd2c67b5c061ff0f4e8ff3d3a176c3f81375458d862"}
Nov 03 20:21:23.690929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:23.690893    5424 generic.go:356] "Generic (PLEG): container finished" podID="5c174cb2-8f31-40e9-b54c-c6e3c14daaca" containerID="5fd4bd52660672dc9c846e85e540ce074106a56082c94c779818e49b3d01e106" exitCode=0
Nov 03 20:21:23.691322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:23.690977    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj" event={"ID":"5c174cb2-8f31-40e9-b54c-c6e3c14daaca","Type":"ContainerDied","Data":"5fd4bd52660672dc9c846e85e540ce074106a56082c94c779818e49b3d01e106"}
Nov 03 20:21:23.871946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:23.871884    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:24.017582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.017554    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\" (UniqueName: \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\") pod \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") "
Nov 03 20:21:24.017758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.017649    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\") pod \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") "
Nov 03 20:21:24.017844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.017822    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\") pod \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\" (UID: \"886aecdc-b29b-40de-ab51-e1ec518bdbf5\") "
Nov 03 20:21:24.018040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.018012    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca" pod "886aecdc-b29b-40de-ab51-e1ec518bdbf5" (UID: "886aecdc-b29b-40de-ab51-e1ec518bdbf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:24.018614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.018558    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:24.020113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.020086    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca" pod "886aecdc-b29b-40de-ab51-e1ec518bdbf5" (UID: "886aecdc-b29b-40de-ab51-e1ec518bdbf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:24.020198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.020137    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj" pod "886aecdc-b29b-40de-ab51-e1ec518bdbf5" (UID: "886aecdc-b29b-40de-ab51-e1ec518bdbf5"). InnerVolumeSpecName "kube-api-access-6svwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:24.120102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.120065    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886aecdc-b29b-40de-ab51-e1ec518bdbf5-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:24.120102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.120099    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svwj\" (UniqueName: \"kubernetes.io/projected/886aecdc-b29b-40de-ab51-e1ec518bdbf5-kube-api-access-6svwj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:24.700548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.700502    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc" event={"ID":"886aecdc-b29b-40de-ab51-e1ec518bdbf5","Type":"ContainerDied","Data":"402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975"}
Nov 03 20:21:24.700941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.700563    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"
Nov 03 20:21:24.700941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.700575    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402a19a5563132c47032723f4ac8127874240cacf10ee583117e0273a3880975"
Nov 03 20:21:24.883159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:24.883136    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:25.046978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.046936    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\" (UniqueName: \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\") pod \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") "
Nov 03 20:21:25.047164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.046989    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\") pod \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") "
Nov 03 20:21:25.047348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.047326    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\") pod \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\" (UID: \"5c174cb2-8f31-40e9-b54c-c6e3c14daaca\") "
Nov 03 20:21:25.047426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.047408    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca" pod "5c174cb2-8f31-40e9-b54c-c6e3c14daaca" (UID: "5c174cb2-8f31-40e9-b54c-c6e3c14daaca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:25.048134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.048110    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.049356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.049331    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27" pod "5c174cb2-8f31-40e9-b54c-c6e3c14daaca" (UID: "5c174cb2-8f31-40e9-b54c-c6e3c14daaca"). InnerVolumeSpecName "kube-api-access-9xv27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:25.049468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.049419    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca" pod "5c174cb2-8f31-40e9-b54c-c6e3c14daaca" (UID: "5c174cb2-8f31-40e9-b54c-c6e3c14daaca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:25.149366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.149333    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.149366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.149363    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xv27\" (UniqueName: \"kubernetes.io/projected/5c174cb2-8f31-40e9-b54c-c6e3c14daaca-kube-api-access-9xv27\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.361025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.360943    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-eece67"]
Nov 03 20:21:25.361632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361612    5424 cpu_manager.go:414] "containerMap: removing container" podUID="886aecdc-b29b-40de-ab51-e1ec518bdbf5" containerName="revoke-keys"
Nov 03 20:21:25.361703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361636    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886aecdc-b29b-40de-ab51-e1ec518bdbf5" containerName="revoke-keys"
Nov 03 20:21:25.361703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361662    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5c174cb2-8f31-40e9-b54c-c6e3c14daaca" containerName="revoke-keys"
Nov 03 20:21:25.361703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361669    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c174cb2-8f31-40e9-b54c-c6e3c14daaca" containerName="revoke-keys"
Nov 03 20:21:25.361875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361863    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="886aecdc-b29b-40de-ab51-e1ec518bdbf5" containerName="revoke-keys"
Nov 03 20:21:25.361908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.361882    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c174cb2-8f31-40e9-b54c-c6e3c14daaca" containerName="revoke-keys"
Nov 03 20:21:25.366343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.366319    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:25.378296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.378266    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-eece67"]
Nov 03 20:21:25.380984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.380957    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2hbv\""
Nov 03 20:21:25.556067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.556024    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk2s\" (UniqueName: \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\") pod \"test-curl-7024-eece67\" (UID: \"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb\") " pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:25.657927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.657836    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk2s\" (UniqueName: \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\") pod \"test-curl-7024-eece67\" (UID: \"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb\") " pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:25.683475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.683437    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk2s\" (UniqueName: \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\") pod \"test-curl-7024-eece67\" (UID: \"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb\") " pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:25.712064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.712033    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"
Nov 03 20:21:25.712064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.712049    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj" event={"ID":"5c174cb2-8f31-40e9-b54c-c6e3c14daaca","Type":"ContainerDied","Data":"ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc"}
Nov 03 20:21:25.712567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.712132    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0ce7f82828f38069196fb87cafb9017a5b60855820259c682bb197768e64dc"
Nov 03 20:21:25.982278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:25.982245    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:26.245548 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:26.245442    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:26.294575 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:26.294510    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded7a6d41_409d_4f4e_9e8b_1fdf782cc4fb.slice/crio-815409d804c17968004e81cfe469797e38749825335615d4a9f62efbfa42c3dc WatchSource:0}: Error finding container 815409d804c17968004e81cfe469797e38749825335615d4a9f62efbfa42c3dc: Status 404 returned error can't find the container with id 815409d804c17968004e81cfe469797e38749825335615d4a9f62efbfa42c3dc
Nov 03 20:21:26.297187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:26.297165    5424 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:21:26.297894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:26.297868    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-eece67"]
Nov 03 20:21:26.722374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:26.722284    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-eece67" event={"ID":"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb","Type":"ContainerStarted","Data":"815409d804c17968004e81cfe469797e38749825335615d4a9f62efbfa42c3dc"}
Nov 03 20:21:28.743955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:28.743919    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-eece67" event={"ID":"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb","Type":"ContainerStarted","Data":"1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4"}
Nov 03 20:21:28.786008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:28.785895    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-eece67" podStartSLOduration=1.580247434 podStartE2EDuration="3.785874985s" totalImagesPullingTime="2.205627551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:28.780827696 +0000 UTC m=+1030.989843696" watchObservedRunningTime="2026-11-03 20:21:28.785874985 +0000 UTC m=+1030.994890995"
Nov 03 20:21:29.226933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.226834    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"]
Nov 03 20:21:29.231216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.231197    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.248632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.248604    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"]
Nov 03 20:21:29.257941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.257903    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:21:29.258138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.258074    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:29.258213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.258166    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:29.258266 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.258236    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:21:29.315461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.315430    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.315676 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.315514    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.315967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.315942    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqml\" (UniqueName: \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.417265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.417235    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.417607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.417583    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqml\" (UniqueName: \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.417696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.417659    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.417982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.417960    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.420110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.420087    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.438540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.438503    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqml\" (UniqueName: \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\") pod \"maas-api-revoke-sub-f72b63d2-499rg\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.546621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.546589    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:29.644826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.644751    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-eece67"]
Nov 03 20:21:29.650003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.649975    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-eece67"]
Nov 03 20:21:29.768399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.768364    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-bb8da4"]
Nov 03 20:21:29.768956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.768943    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" containerName="test-curl-7024-eece67"
Nov 03 20:21:29.768995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.768958    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" containerName="test-curl-7024-eece67"
Nov 03 20:21:29.769078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.769069    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" containerName="test-curl-7024-eece67"
Nov 03 20:21:29.773893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.773876    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:29.790134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.790104    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-bb8da4"]
Nov 03 20:21:29.827282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.827201    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttc2x\" (UniqueName: \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\") pod \"test-curl-7024-bb8da4\" (UID: \"90d2674f-35db-4d00-9619-5ae26a3bb6cf\") " pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:29.858078 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:29.858035    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3545e5c2_1832_45d3_a9b4_149568f1b2b6.slice/crio-5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f WatchSource:0}: Error finding container 5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f: Status 404 returned error can't find the container with id 5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f
Nov 03 20:21:29.862341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.862312    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"]
Nov 03 20:21:29.928802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.928774    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttc2x\" (UniqueName: \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\") pod \"test-curl-7024-bb8da4\" (UID: \"90d2674f-35db-4d00-9619-5ae26a3bb6cf\") " pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:29.947408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:29.947379    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttc2x\" (UniqueName: \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\") pod \"test-curl-7024-bb8da4\" (UID: \"90d2674f-35db-4d00-9619-5ae26a3bb6cf\") " pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:30.090138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.090069    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:30.390964 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:30.390920    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d2674f_35db_4d00_9619_5ae26a3bb6cf.slice/crio-ed80721d8fc92473d80cf5479ed3cfff39b9632dfa6f7b9240c9b01d609546fb WatchSource:0}: Error finding container ed80721d8fc92473d80cf5479ed3cfff39b9632dfa6f7b9240c9b01d609546fb: Status 404 returned error can't find the container with id ed80721d8fc92473d80cf5479ed3cfff39b9632dfa6f7b9240c9b01d609546fb
Nov 03 20:21:30.393224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.393206    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-bb8da4"]
Nov 03 20:21:30.770391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.770360    5424 generic.go:356] "Generic (PLEG): container finished" podID="3545e5c2-1832-45d3-a9b4-149568f1b2b6" containerID="aadf449ac4061670e9759665934bfffb22e741aabf5730bf8f9c1d68f68fa9f5" exitCode=0
Nov 03 20:21:30.770769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.770499    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg" event={"ID":"3545e5c2-1832-45d3-a9b4-149568f1b2b6","Type":"ContainerDied","Data":"aadf449ac4061670e9759665934bfffb22e741aabf5730bf8f9c1d68f68fa9f5"}
Nov 03 20:21:30.770769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.770603    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg" event={"ID":"3545e5c2-1832-45d3-a9b4-149568f1b2b6","Type":"ContainerStarted","Data":"5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f"}
Nov 03 20:21:30.772183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.772156    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-bb8da4" event={"ID":"90d2674f-35db-4d00-9619-5ae26a3bb6cf","Type":"ContainerStarted","Data":"fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442"}
Nov 03 20:21:30.772294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.772193    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-bb8da4" event={"ID":"90d2674f-35db-4d00-9619-5ae26a3bb6cf","Type":"ContainerStarted","Data":"ed80721d8fc92473d80cf5479ed3cfff39b9632dfa6f7b9240c9b01d609546fb"}
Nov 03 20:21:30.772294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.772208    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-eece67" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" containerName="test-curl-7024-eece67" containerID="cri-o://1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4" gracePeriod=2
Nov 03 20:21:30.843007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:30.842958    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-bb8da4" podStartSLOduration=1.842944151 podStartE2EDuration="1.842944151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:30.842493945 +0000 UTC m=+1033.051509942" watchObservedRunningTime="2026-11-03 20:21:30.842944151 +0000 UTC m=+1033.051960147"
Nov 03 20:21:31.292362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.292321    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"]
Nov 03 20:21:31.296791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.296774    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.313897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.313872    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"]
Nov 03 20:21:31.350341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.350308    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.350579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.350552    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.350738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.350721    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdp6\" (UniqueName: \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.452265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.452230    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.452451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.452385    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.452558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.452518    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdp6\" (UniqueName: \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.453142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.453048    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.454844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.454820    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.472343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.472313    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdp6\" (UniqueName: \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\") pod \"maas-api-revoke-sub-82cad2b0-hjzs2\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.612371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.612299    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:31.649837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.649802    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-bb8da4"]
Nov 03 20:21:31.656715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.656682    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-bb8da4"]
Nov 03 20:21:31.919048 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:31.919014    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c28a45_1fba_4ead_9d40_4c64aed8eef3.slice/crio-b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8 WatchSource:0}: Error finding container b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8: Status 404 returned error can't find the container with id b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8
Nov 03 20:21:31.924145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:31.924120    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"]
Nov 03 20:21:32.036491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.036460    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:32.161845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.161813    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\") pod \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") "
Nov 03 20:21:32.162004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.161907    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\") pod \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") "
Nov 03 20:21:32.162004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.161970    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\" (UniqueName: \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\") pod \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\" (UID: \"3545e5c2-1832-45d3-a9b4-149568f1b2b6\") "
Nov 03 20:21:32.162256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.162229    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca" pod "3545e5c2-1832-45d3-a9b4-149568f1b2b6" (UID: "3545e5c2-1832-45d3-a9b4-149568f1b2b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:32.162586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.162566    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:32.164148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.164119    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca" pod "3545e5c2-1832-45d3-a9b4-149568f1b2b6" (UID: "3545e5c2-1832-45d3-a9b4-149568f1b2b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:32.164234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.164191    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml" pod "3545e5c2-1832-45d3-a9b4-149568f1b2b6" (UID: "3545e5c2-1832-45d3-a9b4-149568f1b2b6"). InnerVolumeSpecName "kube-api-access-lbqml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:32.265195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.265152    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3545e5c2-1832-45d3-a9b4-149568f1b2b6-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:32.265358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.265193    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqml\" (UniqueName: \"kubernetes.io/projected/3545e5c2-1832-45d3-a9b4-149568f1b2b6-kube-api-access-lbqml\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:32.792478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.792453    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"
Nov 03 20:21:32.792646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.792488    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg" event={"ID":"3545e5c2-1832-45d3-a9b4-149568f1b2b6","Type":"ContainerDied","Data":"5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f"}
Nov 03 20:21:32.792646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.792626    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed98424e51fd9ccdcfe1043bbf801adf72c8d20630a9bb2869e174c74d7dd6f"
Nov 03 20:21:32.794738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.794708    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2" event={"ID":"d3c28a45-1fba-4ead-9d40-4c64aed8eef3","Type":"ContainerStarted","Data":"f26dc6620968cbd5e4e531f81e85d6b3d14878b45dda2837f3586859bcddabae"}
Nov 03 20:21:32.794842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.794764    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2" event={"ID":"d3c28a45-1fba-4ead-9d40-4c64aed8eef3","Type":"ContainerStarted","Data":"b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8"}
Nov 03 20:21:32.794842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.794812    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-bb8da4" podUID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" containerName="test-curl-7024-bb8da4" containerID="cri-o://fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442" gracePeriod=2
Nov 03 20:21:32.839100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:32.839044    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2" podStartSLOduration=1.839024635 podStartE2EDuration="1.839024635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:32.836152585 +0000 UTC m=+1035.045168607" watchObservedRunningTime="2026-11-03 20:21:32.839024635 +0000 UTC m=+1035.048040632"
Nov 03 20:21:33.013272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.013249    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:33.075520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.075493    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\" (UniqueName: \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\") pod \"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb\" (UID: \"ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb\") "
Nov 03 20:21:33.077895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.077874    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s" pod "ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" (UID: "ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb"). InnerVolumeSpecName "kube-api-access-vqk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:33.177392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.177361    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqk2s\" (UniqueName: \"kubernetes.io/projected/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb-kube-api-access-vqk2s\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:33.508019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.507983    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"]
Nov 03 20:21:33.508847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.508831    5424 cpu_manager.go:414] "containerMap: removing container" podUID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" containerName="test-curl-7024-bb8da4"
Nov 03 20:21:33.508907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.508850    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" containerName="test-curl-7024-bb8da4"
Nov 03 20:21:33.508907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.508885    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3545e5c2-1832-45d3-a9b4-149568f1b2b6" containerName="revoke-keys"
Nov 03 20:21:33.508907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.508893    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3545e5c2-1832-45d3-a9b4-149568f1b2b6" containerName="revoke-keys"
Nov 03 20:21:33.509040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.509029    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" containerName="test-curl-7024-bb8da4"
Nov 03 20:21:33.509070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.509048    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3545e5c2-1832-45d3-a9b4-149568f1b2b6" containerName="revoke-keys"
Nov 03 20:21:33.513233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.513215    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.528624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.528593    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"]
Nov 03 20:21:33.585930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.585894    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.586063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.585966    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.586198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.586173    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pnt\" (UniqueName: \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.687761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.687726    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.687926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.687794    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.687970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.687938    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pnt\" (UniqueName: \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.688421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.688395    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.690252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.690232    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.706586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.706549    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pnt\" (UniqueName: \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\") pod \"maas-api-revoke-sub-08df0367-k6xfd\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.803812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.803732    5424 generic.go:356] "Generic (PLEG): container finished" podID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" containerID="1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4" exitCode=137
Nov 03 20:21:33.803812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.803794    5424 scope.go:122] "RemoveContainer" containerID="1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4"
Nov 03 20:21:33.804011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.803798    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:33.805828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.805806    5424 generic.go:356] "Generic (PLEG): container finished" podID="d3c28a45-1fba-4ead-9d40-4c64aed8eef3" containerID="f26dc6620968cbd5e4e531f81e85d6b3d14878b45dda2837f3586859bcddabae" exitCode=0
Nov 03 20:21:33.805976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.805840    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2" event={"ID":"d3c28a45-1fba-4ead-9d40-4c64aed8eef3","Type":"ContainerDied","Data":"f26dc6620968cbd5e4e531f81e85d6b3d14878b45dda2837f3586859bcddabae"}
Nov 03 20:21:33.815874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.815855    5424 scope.go:122] "RemoveContainer" containerID="1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4"
Nov 03 20:21:33.816131 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:33.816112    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4\": container with ID starting with 1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4 not found: ID does not exist" containerID="1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4"
Nov 03 20:21:33.816177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.816140    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4"} err="failed to get container status \"1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4\": rpc error: code = NotFound desc = could not find container \"1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4\": container with ID starting with 1dbe9517ce0a1a70f4fcf315c2192d3ef5be3d498e08b9c41e73afc1278445b4 not found: ID does not exist"
Nov 03 20:21:33.829598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.829576    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:33.859510 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:33.859326    5424 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7024-eece67\" is forbidden: User \"system:node:ip-10-0-136-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-244.ec2.internal' and this object" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" pod="opendatahub/test-curl-7024-eece67"
Nov 03 20:21:33.927107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:33.927080    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb" path="/var/lib/kubelet/pods/ed7a6d41-409d-4f4e-9e8b-1fdf782cc4fb/volumes"
Nov 03 20:21:34.154089 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:34.154045    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a75021_8b55_4072_b01e_f013dd058674.slice/crio-e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395 WatchSource:0}: Error finding container e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395: Status 404 returned error can't find the container with id e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395
Nov 03 20:21:34.156678 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:34.156654    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"]
Nov 03 20:21:34.903563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:34.903073    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd" event={"ID":"e9a75021-8b55-4072-b01e-f013dd058674","Type":"ContainerStarted","Data":"ff38ca677922c3cda27488196a3c06c1cb23502c25ab7fad8f6f6e8eca568030"}
Nov 03 20:21:34.903563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:34.903145    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd" event={"ID":"e9a75021-8b55-4072-b01e-f013dd058674","Type":"ContainerStarted","Data":"e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395"}
Nov 03 20:21:35.172994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.172947    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:35.193163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.193135    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:35.218575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.218490    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd" podStartSLOduration=2.218469163 podStartE2EDuration="2.218469163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:34.962155045 +0000 UTC m=+1037.171171044" watchObservedRunningTime="2026-11-03 20:21:35.218469163 +0000 UTC m=+1037.427485163"
Nov 03 20:21:35.237063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.237031    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\" (UniqueName: \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\") pod \"90d2674f-35db-4d00-9619-5ae26a3bb6cf\" (UID: \"90d2674f-35db-4d00-9619-5ae26a3bb6cf\") "
Nov 03 20:21:35.237757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.237732    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\" (UniqueName: \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\") pod \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") "
Nov 03 20:21:35.238572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.238059    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\") pod \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") "
Nov 03 20:21:35.238804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.238784    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\") pod \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\" (UID: \"d3c28a45-1fba-4ead-9d40-4c64aed8eef3\") "
Nov 03 20:21:35.243650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.243599    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca" pod "d3c28a45-1fba-4ead-9d40-4c64aed8eef3" (UID: "d3c28a45-1fba-4ead-9d40-4c64aed8eef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:35.244101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.244073    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x" pod "90d2674f-35db-4d00-9619-5ae26a3bb6cf" (UID: "90d2674f-35db-4d00-9619-5ae26a3bb6cf"). InnerVolumeSpecName "kube-api-access-ttc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:35.244416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.244389    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca" pod "d3c28a45-1fba-4ead-9d40-4c64aed8eef3" (UID: "d3c28a45-1fba-4ead-9d40-4c64aed8eef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:35.246671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.246631    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6" pod "d3c28a45-1fba-4ead-9d40-4c64aed8eef3" (UID: "d3c28a45-1fba-4ead-9d40-4c64aed8eef3"). InnerVolumeSpecName "kube-api-access-2bdp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:35.340785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.340694    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:35.340785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.340725    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttc2x\" (UniqueName: \"kubernetes.io/projected/90d2674f-35db-4d00-9619-5ae26a3bb6cf-kube-api-access-ttc2x\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:35.340785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.340738    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdp6\" (UniqueName: \"kubernetes.io/projected/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-kube-api-access-2bdp6\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:35.340785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.340750    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c28a45-1fba-4ead-9d40-4c64aed8eef3-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:35.912252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.912225    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"
Nov 03 20:21:35.914011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.913977    5424 generic.go:356] "Generic (PLEG): container finished" podID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" containerID="fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442" exitCode=137
Nov 03 20:21:35.914172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.914103    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-bb8da4"
Nov 03 20:21:35.916427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.916401    5424 generic.go:356] "Generic (PLEG): container finished" podID="e9a75021-8b55-4072-b01e-f013dd058674" containerID="ff38ca677922c3cda27488196a3c06c1cb23502c25ab7fad8f6f6e8eca568030" exitCode=0
Nov 03 20:21:35.924867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.923155    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d2674f-35db-4d00-9619-5ae26a3bb6cf" path="/var/lib/kubelet/pods/90d2674f-35db-4d00-9619-5ae26a3bb6cf/volumes"
Nov 03 20:21:35.924867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.924071    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2" event={"ID":"d3c28a45-1fba-4ead-9d40-4c64aed8eef3","Type":"ContainerDied","Data":"b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8"}
Nov 03 20:21:35.924867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.924136    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73f67eb065f2338a559b667240a48e832617963a244e99eadbd5eb6ba4052b8"
Nov 03 20:21:35.924867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.924164    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd" event={"ID":"e9a75021-8b55-4072-b01e-f013dd058674","Type":"ContainerDied","Data":"ff38ca677922c3cda27488196a3c06c1cb23502c25ab7fad8f6f6e8eca568030"}
Nov 03 20:21:35.924867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.924208    5424 scope.go:122] "RemoveContainer" containerID="fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442"
Nov 03 20:21:35.940490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.940470    5424 scope.go:122] "RemoveContainer" containerID="fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442"
Nov 03 20:21:35.940824 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:35.940803    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442\": container with ID starting with fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442 not found: ID does not exist" containerID="fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442"
Nov 03 20:21:35.940892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:35.940839    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442"} err="failed to get container status \"fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442\": rpc error: code = NotFound desc = could not find container \"fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442\": container with ID starting with fa3c9e9d4ab167494e2136f8356b030a7324596753e6f49f06ede8ed34bf9442 not found: ID does not exist"
Nov 03 20:21:36.335150 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:36.335093    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:37.084806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.084778    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:37.105878 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:37.105846    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:37.179680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.179592    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\") pod \"e9a75021-8b55-4072-b01e-f013dd058674\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") "
Nov 03 20:21:37.179680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.179663    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\") pod \"e9a75021-8b55-4072-b01e-f013dd058674\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") "
Nov 03 20:21:37.179918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.179829    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\" (UniqueName: \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\") pod \"e9a75021-8b55-4072-b01e-f013dd058674\" (UID: \"e9a75021-8b55-4072-b01e-f013dd058674\") "
Nov 03 20:21:37.187569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.180194    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca" pod "e9a75021-8b55-4072-b01e-f013dd058674" (UID: "e9a75021-8b55-4072-b01e-f013dd058674"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:37.187569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.180614    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:37.187569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.181898    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca" pod "e9a75021-8b55-4072-b01e-f013dd058674" (UID: "e9a75021-8b55-4072-b01e-f013dd058674"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:37.187569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.182402    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt" pod "e9a75021-8b55-4072-b01e-f013dd058674" (UID: "e9a75021-8b55-4072-b01e-f013dd058674"). InnerVolumeSpecName "kube-api-access-f2pnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:37.281613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.281586    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9a75021-8b55-4072-b01e-f013dd058674-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:37.281613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.281611    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pnt\" (UniqueName: \"kubernetes.io/projected/e9a75021-8b55-4072-b01e-f013dd058674-kube-api-access-f2pnt\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:37.939829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.939798    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"
Nov 03 20:21:37.940354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.939834    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd" event={"ID":"e9a75021-8b55-4072-b01e-f013dd058674","Type":"ContainerDied","Data":"e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395"}
Nov 03 20:21:37.940354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:37.939892    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d2716b5b578c4a5467963d74a2734b97ae0b361ccfd72b5819c5dc12c37395"
Nov 03 20:21:39.564148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564118    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"]
Nov 03 20:21:39.564714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564700    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e9a75021-8b55-4072-b01e-f013dd058674" containerName="revoke-keys"
Nov 03 20:21:39.564714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564715    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a75021-8b55-4072-b01e-f013dd058674" containerName="revoke-keys"
Nov 03 20:21:39.564804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564758    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d3c28a45-1fba-4ead-9d40-4c64aed8eef3" containerName="revoke-keys"
Nov 03 20:21:39.564804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564763    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c28a45-1fba-4ead-9d40-4c64aed8eef3" containerName="revoke-keys"
Nov 03 20:21:39.564876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564867    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a75021-8b55-4072-b01e-f013dd058674" containerName="revoke-keys"
Nov 03 20:21:39.564912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.564883    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c28a45-1fba-4ead-9d40-4c64aed8eef3" containerName="revoke-keys"
Nov 03 20:21:39.568766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.568747    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.587507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.587482    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"]
Nov 03 20:21:39.592742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.592717    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:39.592894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.592866    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:39.618960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.618932    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:21:39.619108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.618994    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:21:39.717745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.717711    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlqc\" (UniqueName: \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.717900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.717848    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.717941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.717923    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.819459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.819368    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlqc\" (UniqueName: \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.820932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.819583    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.820932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.819652    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.820932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.820773    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.827285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.827254    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.840828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.840294    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlqc\" (UniqueName: \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\") pod \"maas-api-revoke-sub-1ebb3fff-xzvp2\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:39.887473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:39.887443    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:40.202702 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:40.202671    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f3a9a7_4303_4241_afbc_39e3dab336e7.slice/crio-42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7 WatchSource:0}: Error finding container 42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7: Status 404 returned error can't find the container with id 42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7
Nov 03 20:21:40.205507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:40.205419    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"]
Nov 03 20:21:40.971467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:40.971364    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2" event={"ID":"a4f3a9a7-4303-4241-afbc-39e3dab336e7","Type":"ContainerStarted","Data":"0cfa21abf151923cade4d99da00c81a9aab040490899bbea8ca91a2e09b88721"}
Nov 03 20:21:40.971467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:40.971431    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2" event={"ID":"a4f3a9a7-4303-4241-afbc-39e3dab336e7","Type":"ContainerStarted","Data":"42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7"}
Nov 03 20:21:41.011615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.011559    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2" podStartSLOduration=2.011540976 podStartE2EDuration="2.011540976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:41.009243652 +0000 UTC m=+1043.218259644" watchObservedRunningTime="2026-11-03 20:21:41.011540976 +0000 UTC m=+1043.220556972"
Nov 03 20:21:41.525173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.525127    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:21:41.530844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.530817    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.550411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.546967    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:21:41.551304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.551279    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-11a7d765-openshift-default-dockercfg-bt26q\""
Nov 03 20:21:41.712905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.712872    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.714353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.714327    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.714618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.714601    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.714779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.714764    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzn29\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.714966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.714951    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.715101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.715089    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.715307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.715291    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.715453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.715436    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.715734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.715718    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820514    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820594    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820630    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820674    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzn29\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820746    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820777    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820864    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820884    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.821616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.820991    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.826972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.826863    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.827926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.827897    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.828146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.828123    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.828372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.828348    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.828635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.828608    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.831602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.831459    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.839445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.839416    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.849512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.849472    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.850244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.850216    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzn29\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\") pod \"e2e-ait-11a7d765-openshift-default-775d47679c-6d28f\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") " pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:41.856031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:41.855823    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:42.021978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.021926    5424 generic.go:356] "Generic (PLEG): container finished" podID="a4f3a9a7-4303-4241-afbc-39e3dab336e7" containerID="0cfa21abf151923cade4d99da00c81a9aab040490899bbea8ca91a2e09b88721" exitCode=0
Nov 03 20:21:42.024881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.021989    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2" event={"ID":"a4f3a9a7-4303-4241-afbc-39e3dab336e7","Type":"ContainerDied","Data":"0cfa21abf151923cade4d99da00c81a9aab040490899bbea8ca91a2e09b88721"}
Nov 03 20:21:42.248089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.248058    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:21:42.248864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.248791    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:42.249025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.248886    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:42.249025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:42.248944    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:42.297589 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:42.297501    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:21:43.065661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.053128    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" event={"ID":"33d5bb95-f63d-44ba-bfec-34481b8f12e0","Type":"ContainerStarted","Data":"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"}
Nov 03 20:21:43.065661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.053185    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" event={"ID":"33d5bb95-f63d-44ba-bfec-34481b8f12e0","Type":"ContainerStarted","Data":"677e3d8665e55fae7948d50094c991d76215706dae6661a950083504b981db8b"}
Nov 03 20:21:43.152559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.145152    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" podStartSLOduration=2.145124925 podStartE2EDuration="2.145124925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:43.115568545 +0000 UTC m=+1045.324584560" watchObservedRunningTime="2026-11-03 20:21:43.145124925 +0000 UTC m=+1045.354140925"
Nov 03 20:21:43.328706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.323592    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:21:43.359036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.359008    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.405144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.404424    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9798ad8e-dockercfg-d5fsr\""
Nov 03 20:21:43.420564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.415700    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.420564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.415878    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbz8n\" (UniqueName: \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.439495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.436864    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9798ad8e\""
Nov 03 20:21:43.478565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.470633    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:21:43.594873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.594838    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbz8n\" (UniqueName: \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.595964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.595936    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.609550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.597116    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.803563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.726222    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbz8n\" (UniqueName: \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\") pod \"payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.803563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.726288    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:21:43.803563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.731813    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:43.931381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.896260    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:43.931381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.925967    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:43.963557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:43.933468    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxhx\" (UniqueName: \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.162187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.144728    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.325552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.258509    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxhx\" (UniqueName: \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.325552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.258642    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.325552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.259703    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.325552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.312685    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.352219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.327162    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.352219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.328459    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.534879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.518235    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxhx\" (UniqueName: \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\") pod \"payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") " pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.534879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.518343    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.549083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.548810    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:44.660571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.615234    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.660571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.631790    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.660571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.631916    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.660571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.637647    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.693558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.690362    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:44.693558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.690427    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:21:44.693558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.690499    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:44.738865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.717593    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:21:44.738865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.719348    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.738865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.737646    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.841464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.830945    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:44.946558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.946112    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.955558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.953480    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\") pod \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") "
Nov 03 20:21:44.955558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.953583    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\" (UniqueName: \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\") pod \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") "
Nov 03 20:21:44.955558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.953818    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\") pod \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\" (UID: \"a4f3a9a7-4303-4241-afbc-39e3dab336e7\") "
Nov 03 20:21:44.981569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.964631    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.981569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:44.979739    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca" pod "a4f3a9a7-4303-4241-afbc-39e3dab336e7" (UID: "a4f3a9a7-4303-4241-afbc-39e3dab336e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:45.118673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.087939    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.118673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.089610    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca" pod "a4f3a9a7-4303-4241-afbc-39e3dab336e7" (UID: "a4f3a9a7-4303-4241-afbc-39e3dab336e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:45.118673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.090975    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:45.118673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.091001    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4f3a9a7-4303-4241-afbc-39e3dab336e7-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:45.133936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.133880    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc" pod "a4f3a9a7-4303-4241-afbc-39e3dab336e7" (UID: "a4f3a9a7-4303-4241-afbc-39e3dab336e7"). InnerVolumeSpecName "kube-api-access-2jlqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:45.204459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.204416    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jlqc\" (UniqueName: \"kubernetes.io/projected/a4f3a9a7-4303-4241-afbc-39e3dab336e7-kube-api-access-2jlqc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:45.238163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.236940    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.314701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.314655    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.315755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.315730    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:21:45.495614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.472740    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.495614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.475413    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:21:45.518713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.518666    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.542544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.542491    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:21:45.670589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.667692    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.670589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.668246    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.670589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.668745    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:21:45.670589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.668914    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:21:45.671406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.671377    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.873618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.730087    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.238859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.730122    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.239627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.789611    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.239856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.856418    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.240041 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.967499    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:21:46.240268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.240252    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:21:46.240376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.130123    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.240546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:45.967642    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.240756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.208832    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.374799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.374761    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.391099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.391053    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.746147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.391369    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.746521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.392545    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.747349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.449964    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:21:46.751554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.500148    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.751787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.554627    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.687690    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.688315    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.656893    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.703179    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.703209    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.796558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.719654    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"
Nov 03 20:21:46.916822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:46.873637    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.245260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.244079    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.245260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.244571    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.245260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.244918    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.245260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.245220    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.255033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.253821    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.280577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.262010    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.282687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.243711    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.343576 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:47.330633    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca816181_83ac_44a1_9ac1_617a1dfbf30b.slice/crio-68031b79490d0c93828b362957c63637167c5ac66a2ed58df53ae6194c36b574 WatchSource:0}: Error finding container 68031b79490d0c93828b362957c63637167c5ac66a2ed58df53ae6194c36b574: Status 404 returned error can't find the container with id 68031b79490d0c93828b362957c63637167c5ac66a2ed58df53ae6194c36b574
Nov 03 20:21:47.630615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.480846    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.942646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.518996    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.589806    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" podUID="2cb0b7a8-707d-4413-b244-3e2f313b0e48" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.751947    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.752000    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.752021    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.752052    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.762317    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.762351    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.950857    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.025588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:47.994670    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.260893    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.260982    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.261036    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.261077    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.261117    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.339097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.280437    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.448559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.358215    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.643182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.520878    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.643182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.523720    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" podUID="2cb0b7a8-707d-4413-b244-3e2f313b0e48" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.713689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.713639    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.909637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.908818    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:48.909637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:48.908882    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.178830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.178792    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 20:21:49.201346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.201304    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.232609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.226400    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.323893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.310817    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.323893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.301673    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.348558 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:49.335513    5424 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.393s"
Nov 03 20:21:49.643689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.643654    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.644350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.301737    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.302155    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.302207    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.299675    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.301714    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.640812    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.641829    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.643780    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2" event={"ID":"a4f3a9a7-4303-4241-afbc-39e3dab336e7","Type":"ContainerDied","Data":"42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7"}
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648629    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ef683363746ff7689b41cb5b5f9f015bed06631aeab09b9d8fdf5388bd2bf7"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648699    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648740    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648775    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648811    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648828    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" event={"ID":"ca816181-83ac-44a1-9ac1-617a1dfbf30b","Type":"ContainerStarted","Data":"68031b79490d0c93828b362957c63637167c5ac66a2ed58df53ae6194c36b574"}
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.648883    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:21:49.655553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.649092    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:49.661555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.658104    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.670555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.302180    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.691967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.691930    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.692237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.692206    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.692417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.692399    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.692565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.692525    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.692738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.692721    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.692846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.692826    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.693788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.693760    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" event={"ID":"ef33c3da-1def-4d55-b884-36cf8c93de33","Type":"ContainerStarted","Data":"a360ea401756ea88bac62906662aa63e0d50b0425cd79836e88b7059bddd0131"}
Nov 03 20:21:49.706863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.705924    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.710660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.710604    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.753582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.746672    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.770088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.770049    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.796574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.750697    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.989766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:49.989701    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.308948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.286993    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.308948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.287074    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.308948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.287317    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.493555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.492654    5424 trace.go:236] Trace[1992049527]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-nscg8" (03-Nov-2026 20:21:49.202) (total time: 1289ms):
Nov 03 20:21:50.493555 ip-10-0-136-244 kubenswrapper[5424]: Trace[1992049527]: [1.289858506s] [1.289858506s] END
Nov 03 20:21:50.575653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.570433    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.659494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.648284    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.659494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.648550    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.681606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.679683    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.681606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.680866    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.681606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.681156    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.681606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.681392    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.681992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.681628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.681992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.681874    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.682328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.682248    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.682635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.682480    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.719557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.694773    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.757557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:50.749644    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.000254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.000207    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.113987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.113938    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.114639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.114598    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.201895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.201515    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.368640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.337090    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.368640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.337159    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.368640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.319136    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.436559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.353808    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.503192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.488033    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.648709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.648657    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.649116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.649089    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.649865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.649839    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.653617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.653591    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.826382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693728    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.826811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693768    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.827066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693803    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.827307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693826    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.827562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693855    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.827967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693880    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.828212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693904    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.843555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.693924    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.844894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.769109    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.845802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.769166    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.930265 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:51.930227    5424 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.014s"
Nov 03 20:21:51.942157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.942118    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" event={"ID":"ca816181-83ac-44a1-9ac1-617a1dfbf30b","Type":"ContainerStarted","Data":"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"}
Nov 03 20:21:51.942483 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:51.942460    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:21:52.057809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.057754    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.232495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.232454    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.320247 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:52.314267    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:52.391485 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:52.391447    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.547869    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.547955    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.547987    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.548020    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.548064    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.548093    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.548113    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:21:52.548548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.548149    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"]
Nov 03 20:21:52.549361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.549001    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a4f3a9a7-4303-4241-afbc-39e3dab336e7" containerName="revoke-keys"
Nov 03 20:21:52.549361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.549020    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f3a9a7-4303-4241-afbc-39e3dab336e7" containerName="revoke-keys"
Nov 03 20:21:52.549361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.549182    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f3a9a7-4303-4241-afbc-39e3dab336e7" containerName="revoke-keys"
Nov 03 20:21:52.564773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.564746    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"]
Nov 03 20:21:52.567714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.567692    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:21:52.567925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.567912    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:21:52.568058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.568047    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:21:52.568177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.568167    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:21:52.568286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.568275    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:21:52.568438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.568422    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.570946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.570920    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:21:52.577640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.577509    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:21:52.577840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.577827    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:21:52.578063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.578053    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:21:52.578271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.578261    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:21:52.592074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.592049    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:52.592218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.592202    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:21:52.592294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.592281    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:21:52.592343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.592314    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:52.593291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.593255    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" podStartSLOduration=9.593243596 podStartE2EDuration="9.593243596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:51.500668697 +0000 UTC m=+1053.709684692" watchObservedRunningTime="2026-11-03 20:21:52.593243596 +0000 UTC m=+1054.802259593"
Nov 03 20:21:52.648154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.648121    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.648556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.648540    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rw9\" (UniqueName: \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.648648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.648597    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.750016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.749981    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rw9\" (UniqueName: \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.750178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.750050    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.750236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.750217    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.751166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.751109    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.753197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.753167    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.770046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.770012    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rw9\" (UniqueName: \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\") pod \"maas-api-revoke-sub-648e0dcf-f2x7t\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:52.898575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:52.898457    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:53.234456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.234427    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"]
Nov 03 20:21:53.239231 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:53.239193    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a38ebf_e42c_4ba5_84f2_951e50c47ff5.slice/crio-d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a WatchSource:0}: Error finding container d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a: Status 404 returned error can't find the container with id d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a
Nov 03 20:21:53.331544 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:53.331489    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:53.339688 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:21:53.337966    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:53.594356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.594317    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t" event={"ID":"84a38ebf-e42c-4ba5-84f2-951e50c47ff5","Type":"ContainerStarted","Data":"bfa87f420c8af8e0c4775518a2c7d28fb7a749e438d346804a0e3b539f11ae4e"}
Nov 03 20:21:53.594850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.594389    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t" event={"ID":"84a38ebf-e42c-4ba5-84f2-951e50c47ff5","Type":"ContainerStarted","Data":"d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a"}
Nov 03 20:21:53.596318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.596282    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" event={"ID":"ef33c3da-1def-4d55-b884-36cf8c93de33","Type":"ContainerStarted","Data":"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"}
Nov 03 20:21:53.596450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.596371    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:21:53.636598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.636545    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t" podStartSLOduration=4.636513827 podStartE2EDuration="4.636513827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:53.632668737 +0000 UTC m=+1055.841684734" watchObservedRunningTime="2026-11-03 20:21:53.636513827 +0000 UTC m=+1055.845529824"
Nov 03 20:21:53.674694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:53.674640    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" podStartSLOduration=10.674626813 podStartE2EDuration="10.674626813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:53.671074328 +0000 UTC m=+1055.880090330" watchObservedRunningTime="2026-11-03 20:21:53.674626813 +0000 UTC m=+1055.883642829"
Nov 03 20:21:54.609751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:54.609643    5424 generic.go:356] "Generic (PLEG): container finished" podID="84a38ebf-e42c-4ba5-84f2-951e50c47ff5" containerID="bfa87f420c8af8e0c4775518a2c7d28fb7a749e438d346804a0e3b539f11ae4e" exitCode=0
Nov 03 20:21:54.610210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:54.609785    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t" event={"ID":"84a38ebf-e42c-4ba5-84f2-951e50c47ff5","Type":"ContainerDied","Data":"bfa87f420c8af8e0c4775518a2c7d28fb7a749e438d346804a0e3b539f11ae4e"}
Nov 03 20:21:55.787394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.787364    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:55.904444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.904370    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\") pod \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") "
Nov 03 20:21:55.904628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.904474    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\" (UniqueName: \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\") pod \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") "
Nov 03 20:21:55.904628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.904499    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\") pod \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\" (UID: \"84a38ebf-e42c-4ba5-84f2-951e50c47ff5\") "
Nov 03 20:21:55.904871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.904848    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca" pod "84a38ebf-e42c-4ba5-84f2-951e50c47ff5" (UID: "84a38ebf-e42c-4ba5-84f2-951e50c47ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:55.905284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.905248    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:55.906771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.906748    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca" pod "84a38ebf-e42c-4ba5-84f2-951e50c47ff5" (UID: "84a38ebf-e42c-4ba5-84f2-951e50c47ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:55.906839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:55.906799    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9" pod "84a38ebf-e42c-4ba5-84f2-951e50c47ff5" (UID: "84a38ebf-e42c-4ba5-84f2-951e50c47ff5"). InnerVolumeSpecName "kube-api-access-b4rw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:56.007190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:56.007144    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rw9\" (UniqueName: \"kubernetes.io/projected/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-kube-api-access-b4rw9\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:56.007190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:56.007184    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a38ebf-e42c-4ba5-84f2-951e50c47ff5-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:21:56.630246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:56.630211    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"
Nov 03 20:21:56.630246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:56.630223    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t" event={"ID":"84a38ebf-e42c-4ba5-84f2-951e50c47ff5","Type":"ContainerDied","Data":"d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a"}
Nov 03 20:21:56.630449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:56.630299    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5cc6abf2516fd1693af3325d94debfa2a88d43271064c14653fbdb657086b0a"
Nov 03 20:21:58.749657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.749617    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:21:58.750647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.750624    5424 cpu_manager.go:414] "containerMap: removing container" podUID="84a38ebf-e42c-4ba5-84f2-951e50c47ff5" containerName="revoke-keys"
Nov 03 20:21:58.750723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.750651    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a38ebf-e42c-4ba5-84f2-951e50c47ff5" containerName="revoke-keys"
Nov 03 20:21:58.750862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.750847    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a38ebf-e42c-4ba5-84f2-951e50c47ff5" containerName="revoke-keys"
Nov 03 20:21:58.755603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.755577    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:58.771298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.771251    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:21:58.854553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.843854    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-11a7d765-dockercfg-hkttc\""
Nov 03 20:21:58.854787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.844277    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-11a7d765\""
Nov 03 20:21:58.867648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.867612    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxfz\" (UniqueName: \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:58.867909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.867891    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:58.979078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.977230    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxfz\" (UniqueName: \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:58.979078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.977311    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:58.979078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:58.978336    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:59.003729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.003650    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxfz\" (UniqueName: \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\") pod \"payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:59.077538 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.077490    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:59.406067 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:21:59.406034    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c509a3c_56b3_4cc4_943b_eb013f9a794f.slice/crio-ad2f1c0525d282bf2b1863c8026f110addcdd54fc6008ac5a4c73ea0d7befcb8 WatchSource:0}: Error finding container ad2f1c0525d282bf2b1863c8026f110addcdd54fc6008ac5a4c73ea0d7befcb8: Status 404 returned error can't find the container with id ad2f1c0525d282bf2b1863c8026f110addcdd54fc6008ac5a4c73ea0d7befcb8
Nov 03 20:21:59.409056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.408976    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:21:59.658129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.658052    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" event={"ID":"6c509a3c-56b3-4cc4-943b-eb013f9a794f","Type":"ContainerStarted","Data":"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"}
Nov 03 20:21:59.658129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.658108    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" event={"ID":"6c509a3c-56b3-4cc4-943b-eb013f9a794f","Type":"ContainerStarted","Data":"ad2f1c0525d282bf2b1863c8026f110addcdd54fc6008ac5a4c73ea0d7befcb8"}
Nov 03 20:21:59.658320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.658139    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:21:59.703424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:21:59.703368    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" podStartSLOduration=1.703350188 podStartE2EDuration="1.703350188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:59.698725205 +0000 UTC m=+1061.907741205" watchObservedRunningTime="2026-11-03 20:21:59.703350188 +0000 UTC m=+1061.912366185"
Nov 03 20:22:02.569493 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:02.569449    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:22:03.599726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:03.599691    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:22:04.613565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.613517    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:22:04.713433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.713401    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:22:04.718599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.718570    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.737884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.737856    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b67df53-openshift-default-dockercfg-fhcht\""
Nov 03 20:22:04.744361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.744338    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:22:04.775579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.775521    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rktn7\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.775854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.775830    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.775998    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776203    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776370    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776503    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776687    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.776842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776826    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.777015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.776999    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.889945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.889804    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rktn7\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.903917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.903321    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.904194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.903874    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.922333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.904329    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.922782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.922738    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.930728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.930700    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.930878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.930753    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.930878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.930814    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.930878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.930863    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.931048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.930954    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.931104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.931050    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.931766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.931681    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.941452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.940005    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.949586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.942017    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rktn7\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.959558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.958447    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.984553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.968646    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.985906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.985873    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:04.986117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:04.986095    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\") pod \"e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") " pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:05.043390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.043360    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:05.418566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.418497    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:05.418748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.418730    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:05.418892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.418876    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:05.420350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.420322    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:22:05.725245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.725141    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" event={"ID":"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0","Type":"ContainerStarted","Data":"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"}
Nov 03 20:22:05.725245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.725236    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" event={"ID":"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0","Type":"ContainerStarted","Data":"b8166774703f221f172593e7558108171252b7ca8620f6cc353271454c540094"}
Nov 03 20:22:05.800805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:05.800743    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" podStartSLOduration=1.80072795 podStartE2EDuration="1.80072795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:05.796081435 +0000 UTC m=+1068.005097439" watchObservedRunningTime="2026-11-03 20:22:05.80072795 +0000 UTC m=+1068.009743947"
Nov 03 20:22:06.044857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:06.044824    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:06.046477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:06.046443    5424 patch_prober.go:29] interesting pod/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:22:06.046644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:06.046572    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 20:22:07.049658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.049628    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:07.120704 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:07.120665    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache]"
Nov 03 20:22:07.381550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.381443    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"]
Nov 03 20:22:07.386218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.386193    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.413960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.412192    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:22:07.413960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.412940    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:22:07.420979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.420949    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"]
Nov 03 20:22:07.425768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.425743    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:22:07.425909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.425861    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:22:07.513508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.513477    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.513765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.513748    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6z5\" (UniqueName: \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.513835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.513806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.615608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.615576    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6z5\" (UniqueName: \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.615835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.615647    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.615835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.615775    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.616384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.616357    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.618280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.618261    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.635081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.635015    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6z5\" (UniqueName: \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\") pod \"maas-api-revoke-sub-6daa7982-frsdr\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.724880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.724847    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:07.744158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.744132    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:07.745297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:07.745274    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:22:08.038584 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:08.038448    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676d2f6f_0288_4a1f_ab92_8dd64aa2295c.slice/crio-5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070 WatchSource:0}: Error finding container 5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070: Status 404 returned error can't find the container with id 5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070
Nov 03 20:22:08.064210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:08.064157    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"]
Nov 03 20:22:08.753355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:08.753317    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr" event={"ID":"676d2f6f-0288-4a1f-ab92-8dd64aa2295c","Type":"ContainerStarted","Data":"cc0383df9ac433f16d1c8267818639209d0c8a32a7b6bb979a0be60ebb68a6e6"}
Nov 03 20:22:08.753576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:08.753411    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr" event={"ID":"676d2f6f-0288-4a1f-ab92-8dd64aa2295c","Type":"ContainerStarted","Data":"5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070"}
Nov 03 20:22:08.794899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:08.794845    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr" podStartSLOduration=1.79483094 podStartE2EDuration="1.79483094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:08.792713635 +0000 UTC m=+1071.001729636" watchObservedRunningTime="2026-11-03 20:22:08.79483094 +0000 UTC m=+1071.003846983"
Nov 03 20:22:09.371767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.371693    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:22:09.376660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.376638    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.391497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.391467    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b67df53\""
Nov 03 20:22:09.391685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.391591    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b67df53-dockercfg-hz5hg\""
Nov 03 20:22:09.393890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.393865    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:22:09.489853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.485810    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.489853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.485937    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79z8m\" (UniqueName: \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.612286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.604931    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.612286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.605067    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79z8m\" (UniqueName: \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.612286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.605832    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.629064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.628987    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79z8m\" (UniqueName: \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\") pod \"payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.692201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.692170    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:09.774159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.774123    5424 generic.go:356] "Generic (PLEG): container finished" podID="676d2f6f-0288-4a1f-ab92-8dd64aa2295c" containerID="cc0383df9ac433f16d1c8267818639209d0c8a32a7b6bb979a0be60ebb68a6e6" exitCode=0
Nov 03 20:22:09.774300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:09.774202    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr" event={"ID":"676d2f6f-0288-4a1f-ab92-8dd64aa2295c","Type":"ContainerDied","Data":"cc0383df9ac433f16d1c8267818639209d0c8a32a7b6bb979a0be60ebb68a6e6"}
Nov 03 20:22:10.004515 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:10.004486    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b18f66c_56ea_4954_9bdb_6562bd36afd4.slice/crio-75858aa034ccd1db1d80fb8653c977a6add2a2a1e231cbbbfb61375b7b53d312 WatchSource:0}: Error finding container 75858aa034ccd1db1d80fb8653c977a6add2a2a1e231cbbbfb61375b7b53d312: Status 404 returned error can't find the container with id 75858aa034ccd1db1d80fb8653c977a6add2a2a1e231cbbbfb61375b7b53d312
Nov 03 20:22:10.007478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.007455    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:22:10.670630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.670604    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:22:10.790833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.790800    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" event={"ID":"5b18f66c-56ea-4954-9bdb-6562bd36afd4","Type":"ContainerStarted","Data":"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"}
Nov 03 20:22:10.791150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.791127    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" event={"ID":"5b18f66c-56ea-4954-9bdb-6562bd36afd4","Type":"ContainerStarted","Data":"75858aa034ccd1db1d80fb8653c977a6add2a2a1e231cbbbfb61375b7b53d312"}
Nov 03 20:22:10.791312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.791296    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:10.991741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:10.991716    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:11.036303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.036241    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" podStartSLOduration=2.036222201 podStartE2EDuration="2.036222201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:10.835169905 +0000 UTC m=+1073.044185902" watchObservedRunningTime="2026-11-03 20:22:11.036222201 +0000 UTC m=+1073.245238198"
Nov 03 20:22:11.041144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.041097    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\") pod \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") "
Nov 03 20:22:11.041314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.041264    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\") pod \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") "
Nov 03 20:22:11.041592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.041513    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\" (UniqueName: \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\") pod \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\" (UID: \"676d2f6f-0288-4a1f-ab92-8dd64aa2295c\") "
Nov 03 20:22:11.043822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.043405    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca" pod "676d2f6f-0288-4a1f-ab92-8dd64aa2295c" (UID: "676d2f6f-0288-4a1f-ab92-8dd64aa2295c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:11.046031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.045991    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca" pod "676d2f6f-0288-4a1f-ab92-8dd64aa2295c" (UID: "676d2f6f-0288-4a1f-ab92-8dd64aa2295c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:11.054834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.054788    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5" pod "676d2f6f-0288-4a1f-ab92-8dd64aa2295c" (UID: "676d2f6f-0288-4a1f-ab92-8dd64aa2295c"). InnerVolumeSpecName "kube-api-access-gb6z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:11.144060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.144028    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6z5\" (UniqueName: \"kubernetes.io/projected/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-kube-api-access-gb6z5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:11.144060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.144056    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:11.144060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.144068    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676d2f6f-0288-4a1f-ab92-8dd64aa2295c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:11.804030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.803997    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"
Nov 03 20:22:11.804459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.804002    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr" event={"ID":"676d2f6f-0288-4a1f-ab92-8dd64aa2295c","Type":"ContainerDied","Data":"5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070"}
Nov 03 20:22:11.804459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:11.804162    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5292d2fe627f3763c51e8ab113a850e6fde880c2e51e122cc46efa8d3df53070"
Nov 03 20:22:12.307469 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:12.307422    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:22:12.642961 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:12.642872    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice/crio-c626a3e5b683213c4c313398e6e6c5f5f813542d06d2b9a8405a195b55ab9537\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13b617e_38de_40e6_9f35_87629d33a97c.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:22:19.536557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.535628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.559558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.559208    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.610559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.609695    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.625570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.616881    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.660564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.660050    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.668717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.668655    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.689715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.689653    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.710583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.710511    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.762767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.762701    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.788764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.788650    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.934554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.931708    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.968570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.966634    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.968570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:19.967048    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.019418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.016726    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.035101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.035064    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:20.035411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.035380    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.261751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.261688    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.262409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.262378    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.591660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.585672    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.591660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.587323    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.633083    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.633496    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.634269    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.634332    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.637618    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.655773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.654739    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:22:20.724601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.661783    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.724601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.665161    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:22:20.724601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.688964    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.724601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.689385    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:22:20.724601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.690962    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.767310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.766238    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.791507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.767621    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:22:20.791507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.767693    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:22:20.803577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.791940    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.803577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.794150    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:22:20.819553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.805195    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.966479    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.966920    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.975641    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.975956    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.976431    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:20.976489    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.083222    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.167492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.084632    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.254233    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.255328    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.267780    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.268076    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.268589    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:22:21.280916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.268642    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:22:21.355557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.355041    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:22:21.412306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.412275    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:22:21.445797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.445709    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:22:21.481749 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.481722    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:22:21.516704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.516679    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:22:21.562314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.560115    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:22:21.571517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.571490    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:22:21.644187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.636867    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:22:21.644187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.636921    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:22:21.696706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.672762    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.696706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.693065    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.852408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.849142    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.896049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.895904    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:21.896049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.895961    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.970645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.966880    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.972488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.971845    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:22:21.972488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:21.971924    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:22:22.133556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.125104    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.459554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.459174    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:22:22.505556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.505063    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:22:22.712672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.712580    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:22:22.722617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.718645    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.773985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:22.773956    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:22:23.607579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.606703    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"]
Nov 03 20:22:23.607808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.607682    5424 cpu_manager.go:414] "containerMap: removing container" podUID="676d2f6f-0288-4a1f-ab92-8dd64aa2295c" containerName="revoke-keys"
Nov 03 20:22:23.607808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.607701    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676d2f6f-0288-4a1f-ab92-8dd64aa2295c" containerName="revoke-keys"
Nov 03 20:22:23.607942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.607927    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="676d2f6f-0288-4a1f-ab92-8dd64aa2295c" containerName="revoke-keys"
Nov 03 20:22:23.614089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.614059    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.629602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.629457    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:22:23.631079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.629686    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:22:23.632163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.632114    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"]
Nov 03 20:22:23.643261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.643098    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:22:23.643485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.643278    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:22:23.749356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.749316    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.749823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.749635    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcfr\" (UniqueName: \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.749823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.749811    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.850936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.850898    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcfr\" (UniqueName: \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.851125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.850991    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.851125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.851116    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.851666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.851643    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.853614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.853589    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.871139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.871055    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcfr\" (UniqueName: \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\") pod \"maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:23.953485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:23.953446    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:24.271164 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:24.271112    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0595821_0dde_4702_b6e3_189a298a1a0d.slice/crio-9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875 WatchSource:0}: Error finding container 9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875: Status 404 returned error can't find the container with id 9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875
Nov 03 20:22:24.273929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:24.273903    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"]
Nov 03 20:22:24.594910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:24.594859    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p" event={"ID":"e0595821-0dde-4702-b6e3-189a298a1a0d","Type":"ContainerStarted","Data":"6833faf51a2cfd9f1c8c1dbf9fb2e0d6f83b5bea24d979bd2c1b97c31d181b20"}
Nov 03 20:22:24.595118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:24.594924    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p" event={"ID":"e0595821-0dde-4702-b6e3-189a298a1a0d","Type":"ContainerStarted","Data":"9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875"}
Nov 03 20:22:24.639971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:24.639919    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p" podStartSLOduration=1.639903419 podStartE2EDuration="1.639903419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:24.633751478 +0000 UTC m=+1086.842767473" watchObservedRunningTime="2026-11-03 20:22:24.639903419 +0000 UTC m=+1086.848919435"
Nov 03 20:22:25.033782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:25.033726    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"]
Nov 03 20:22:25.040612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:25.040580    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43367f1b-6dkfv"]
Nov 03 20:22:25.606298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:25.606261    5424 generic.go:356] "Generic (PLEG): container finished" podID="e0595821-0dde-4702-b6e3-189a298a1a0d" containerID="6833faf51a2cfd9f1c8c1dbf9fb2e0d6f83b5bea24d979bd2c1b97c31d181b20" exitCode=0
Nov 03 20:22:25.606492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:25.606337    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p" event={"ID":"e0595821-0dde-4702-b6e3-189a298a1a0d","Type":"ContainerDied","Data":"6833faf51a2cfd9f1c8c1dbf9fb2e0d6f83b5bea24d979bd2c1b97c31d181b20"}
Nov 03 20:22:25.921776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:25.921691    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603dda62-655d-4d64-98fc-7397cbf81410" path="/var/lib/kubelet/pods/603dda62-655d-4d64-98fc-7397cbf81410/volumes"
Nov 03 20:22:26.868746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.868658    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:26.902881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.902845    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\") pod \"e0595821-0dde-4702-b6e3-189a298a1a0d\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") "
Nov 03 20:22:26.903130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.903055    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\") pod \"e0595821-0dde-4702-b6e3-189a298a1a0d\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") "
Nov 03 20:22:26.903207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.903135    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\" (UniqueName: \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\") pod \"e0595821-0dde-4702-b6e3-189a298a1a0d\" (UID: \"e0595821-0dde-4702-b6e3-189a298a1a0d\") "
Nov 03 20:22:26.903207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.903177    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca" pod "e0595821-0dde-4702-b6e3-189a298a1a0d" (UID: "e0595821-0dde-4702-b6e3-189a298a1a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:26.903920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.903892    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:26.905557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.905509    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr" pod "e0595821-0dde-4702-b6e3-189a298a1a0d" (UID: "e0595821-0dde-4702-b6e3-189a298a1a0d"). InnerVolumeSpecName "kube-api-access-bhcfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:26.905956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:26.905929    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca" pod "e0595821-0dde-4702-b6e3-189a298a1a0d" (UID: "e0595821-0dde-4702-b6e3-189a298a1a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:27.006165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:27.006085    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0595821-0dde-4702-b6e3-189a298a1a0d-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:27.006165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:27.006113    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcfr\" (UniqueName: \"kubernetes.io/projected/e0595821-0dde-4702-b6e3-189a298a1a0d-kube-api-access-bhcfr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:27.676558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:27.665109    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p" event={"ID":"e0595821-0dde-4702-b6e3-189a298a1a0d","Type":"ContainerDied","Data":"9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875"}
Nov 03 20:22:27.676558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:27.665180    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9464ed0ad0384cea152e496ace409331d035595f4811a710ddc469e582bfb875"
Nov 03 20:22:27.676558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:27.665250    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"
Nov 03 20:22:31.821615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.821580    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"]
Nov 03 20:22:31.822226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.822210    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e0595821-0dde-4702-b6e3-189a298a1a0d" containerName="revoke-keys"
Nov 03 20:22:31.822285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.822226    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0595821-0dde-4702-b6e3-189a298a1a0d" containerName="revoke-keys"
Nov 03 20:22:31.822376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.822367    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0595821-0dde-4702-b6e3-189a298a1a0d" containerName="revoke-keys"
Nov 03 20:22:31.826416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.826396    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.844055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.844028    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"]
Nov 03 20:22:31.863774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.849131    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:22:31.863774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.849247    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:22:31.863774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.852640    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:22:31.863774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.853788    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:22:31.895578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.893302    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgh56\" (UniqueName: \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.895578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.893628    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.895578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.893690    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.995214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.995179    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.995390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.995228    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.995390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.995379    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgh56\" (UniqueName: \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.996082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.996056    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:31.997757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:31.997739    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:32.015189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:32.015155    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgh56\" (UniqueName: \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\") pod \"maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:32.143428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:32.143329    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:32.451280 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:32.451244    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b917a0a_36bb_485d_a732_9fc7bc666455.slice/crio-0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085 WatchSource:0}: Error finding container 0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085: Status 404 returned error can't find the container with id 0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085
Nov 03 20:22:32.458662 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:32.458630    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"]
Nov 03 20:22:32.717969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:32.717877    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv" event={"ID":"9b917a0a-36bb-485d-a732-9fc7bc666455","Type":"ContainerStarted","Data":"0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085"}
Nov 03 20:22:33.728000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:33.727910    5424 generic.go:356] "Generic (PLEG): container finished" podID="9b917a0a-36bb-485d-a732-9fc7bc666455" containerID="4c9c9e4b944c47b7c11b08432acaf867d9cd793d72fdab1e76f1f3962cde9efc" exitCode=0
Nov 03 20:22:33.728371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:33.727990    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv" event={"ID":"9b917a0a-36bb-485d-a732-9fc7bc666455","Type":"ContainerDied","Data":"4c9c9e4b944c47b7c11b08432acaf867d9cd793d72fdab1e76f1f3962cde9efc"}
Nov 03 20:22:34.028426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:34.028387    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:22:34.035348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:34.035315    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"]
Nov 03 20:22:34.040504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:34.040479    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65d0869-tzmpc"]
Nov 03 20:22:34.071779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:34.071748    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:22:35.035368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.035340    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:35.151418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.151334    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\") pod \"9b917a0a-36bb-485d-a732-9fc7bc666455\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") "
Nov 03 20:22:35.151418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.151377    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\") pod \"9b917a0a-36bb-485d-a732-9fc7bc666455\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") "
Nov 03 20:22:35.151673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.151488    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\" (UniqueName: \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\") pod \"9b917a0a-36bb-485d-a732-9fc7bc666455\" (UID: \"9b917a0a-36bb-485d-a732-9fc7bc666455\") "
Nov 03 20:22:35.151743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.151717    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca" pod "9b917a0a-36bb-485d-a732-9fc7bc666455" (UID: "9b917a0a-36bb-485d-a732-9fc7bc666455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:35.152106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.152082    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:35.153769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.153738    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca" pod "9b917a0a-36bb-485d-a732-9fc7bc666455" (UID: "9b917a0a-36bb-485d-a732-9fc7bc666455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:35.153897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.153878    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56" pod "9b917a0a-36bb-485d-a732-9fc7bc666455" (UID: "9b917a0a-36bb-485d-a732-9fc7bc666455"). InnerVolumeSpecName "kube-api-access-sgh56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:35.253297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.253259    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b917a0a-36bb-485d-a732-9fc7bc666455-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:35.253297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.253294    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgh56\" (UniqueName: \"kubernetes.io/projected/9b917a0a-36bb-485d-a732-9fc7bc666455-kube-api-access-sgh56\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:35.800652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.800606    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv" event={"ID":"9b917a0a-36bb-485d-a732-9fc7bc666455","Type":"ContainerDied","Data":"0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085"}
Nov 03 20:22:35.800652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.800638    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"
Nov 03 20:22:35.800863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.800662    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee086e3d1af735afa87a23afeaaef9545ae1db851b137b87f4c78098ef7f085"
Nov 03 20:22:35.921159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:35.921126    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df06fb2-feda-4f35-aea7-edc251b7c341" path="/var/lib/kubelet/pods/2df06fb2-feda-4f35-aea7-edc251b7c341/volumes"
Nov 03 20:22:39.029871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.029825    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" podUID="ef33c3da-1def-4d55-b884-36cf8c93de33" containerName="payload-processing" containerID="cri-o://7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121" gracePeriod=25
Nov 03 20:22:39.072948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.072828    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" podUID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" containerName="payload-pre-processing" containerID="cri-o://2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b" gracePeriod=25
Nov 03 20:22:39.166514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.166478    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"]
Nov 03 20:22:39.178272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.178184    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9798ad8e-tbq6p"]
Nov 03 20:22:39.531456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.526670    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:22:39.531456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.529155    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:22:39.641419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.641324    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\" (UniqueName: \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\") pod \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") "
Nov 03 20:22:39.641614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.641579    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\") pod \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\" (UID: \"ca816181-83ac-44a1-9ac1-617a1dfbf30b\") "
Nov 03 20:22:39.641691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.641635    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\") pod \"ef33c3da-1def-4d55-b884-36cf8c93de33\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") "
Nov 03 20:22:39.641803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.641786    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\" (UniqueName: \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\") pod \"ef33c3da-1def-4d55-b884-36cf8c93de33\" (UID: \"ef33c3da-1def-4d55-b884-36cf8c93de33\") "
Nov 03 20:22:39.642154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.642123    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume" pod "ca816181-83ac-44a1-9ac1-617a1dfbf30b" (UID: "ca816181-83ac-44a1-9ac1-617a1dfbf30b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:39.642298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.642168    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume" pod "ef33c3da-1def-4d55-b884-36cf8c93de33" (UID: "ef33c3da-1def-4d55-b884-36cf8c93de33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:39.642632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.642598    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca816181-83ac-44a1-9ac1-617a1dfbf30b-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:39.642632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.642627    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef33c3da-1def-4d55-b884-36cf8c93de33-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:39.645381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.645350    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n" pod "ca816181-83ac-44a1-9ac1-617a1dfbf30b" (UID: "ca816181-83ac-44a1-9ac1-617a1dfbf30b"). InnerVolumeSpecName "kube-api-access-nbz8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:39.645696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.645664    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx" pod "ef33c3da-1def-4d55-b884-36cf8c93de33" (UID: "ef33c3da-1def-4d55-b884-36cf8c93de33"). InnerVolumeSpecName "kube-api-access-vnxhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:39.744587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.744508    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbz8n\" (UniqueName: \"kubernetes.io/projected/ca816181-83ac-44a1-9ac1-617a1dfbf30b-kube-api-access-nbz8n\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:39.744587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.744586    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxhx\" (UniqueName: \"kubernetes.io/projected/ef33c3da-1def-4d55-b884-36cf8c93de33-kube-api-access-vnxhx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:39.870217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.870177    5424 generic.go:356] "Generic (PLEG): container finished" podID="ef33c3da-1def-4d55-b884-36cf8c93de33" containerID="7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121" exitCode=0
Nov 03 20:22:39.870413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.870303    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" event={"ID":"ef33c3da-1def-4d55-b884-36cf8c93de33","Type":"ContainerDied","Data":"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"}
Nov 03 20:22:39.870413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.870358    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868" event={"ID":"ef33c3da-1def-4d55-b884-36cf8c93de33","Type":"ContainerDied","Data":"a360ea401756ea88bac62906662aa63e0d50b0425cd79836e88b7059bddd0131"}
Nov 03 20:22:39.870413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.870388    5424 scope.go:122] "RemoveContainer" containerID="7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"
Nov 03 20:22:39.870608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.870580    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"
Nov 03 20:22:39.878673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.878638    5424 generic.go:356] "Generic (PLEG): container finished" podID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" containerID="2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b" exitCode=0
Nov 03 20:22:39.878921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.878889    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"
Nov 03 20:22:39.879400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.878944    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" event={"ID":"ca816181-83ac-44a1-9ac1-617a1dfbf30b","Type":"ContainerDied","Data":"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"}
Nov 03 20:22:39.879400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.879394    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq" event={"ID":"ca816181-83ac-44a1-9ac1-617a1dfbf30b","Type":"ContainerDied","Data":"68031b79490d0c93828b362957c63637167c5ac66a2ed58df53ae6194c36b574"}
Nov 03 20:22:39.916478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.916451    5424 scope.go:122] "RemoveContainer" containerID="7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"
Nov 03 20:22:39.917206 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:39.917178    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121\": container with ID starting with 7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121 not found: ID does not exist" containerID="7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"
Nov 03 20:22:39.917337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.917218    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121"} err="failed to get container status \"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121\": rpc error: code = NotFound desc = could not find container \"7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121\": container with ID starting with 7b8307a6909ef9063ecfce81801588aa497d393fd6857668db59f6f13c855121 not found: ID does not exist"
Nov 03 20:22:39.917337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.917246    5424 scope.go:122] "RemoveContainer" containerID="2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"
Nov 03 20:22:39.924412 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.924378    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0595821-0dde-4702-b6e3-189a298a1a0d" path="/var/lib/kubelet/pods/e0595821-0dde-4702-b6e3-189a298a1a0d/volumes"
Nov 03 20:22:39.932613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.932589    5424 scope.go:122] "RemoveContainer" containerID="2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"
Nov 03 20:22:39.933025 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:39.932994    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b\": container with ID starting with 2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b not found: ID does not exist" containerID="2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"
Nov 03 20:22:39.933097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.933030    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b"} err="failed to get container status \"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b\": rpc error: code = NotFound desc = could not find container \"2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b\": container with ID starting with 2b33f261bad2a36455ea3539852c8ffab2d3c2af2fb6e40f3b0c02003aac822b not found: ID does not exist"
Nov 03 20:22:39.948909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.948876    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:22:39.955174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.955139    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9798ad8e-758bdb869f-tghnq"]
Nov 03 20:22:39.986196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.986165    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:22:39.992690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:39.992667    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9798ad8e-d5d654d94-hq868"]
Nov 03 20:22:40.333657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.333623    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:22:40.334245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334230    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" containerName="payload-pre-processing"
Nov 03 20:22:40.334288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334247    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" containerName="payload-pre-processing"
Nov 03 20:22:40.334288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334260    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9b917a0a-36bb-485d-a732-9fc7bc666455" containerName="revoke-keys"
Nov 03 20:22:40.334288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334265    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b917a0a-36bb-485d-a732-9fc7bc666455" containerName="revoke-keys"
Nov 03 20:22:40.334288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334276    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ef33c3da-1def-4d55-b884-36cf8c93de33" containerName="payload-processing"
Nov 03 20:22:40.334288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334282    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef33c3da-1def-4d55-b884-36cf8c93de33" containerName="payload-processing"
Nov 03 20:22:40.334441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334408    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" containerName="payload-pre-processing"
Nov 03 20:22:40.334441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334416    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b917a0a-36bb-485d-a732-9fc7bc666455" containerName="revoke-keys"
Nov 03 20:22:40.334441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.334423    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef33c3da-1def-4d55-b884-36cf8c93de33" containerName="payload-processing"
Nov 03 20:22:40.344803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.344763    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.399792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.399761    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-04bf5fc9-openshift-default-dockercfg-5f827\""
Nov 03 20:22:40.415506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.415462    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:22:40.474558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474400    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474464    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474500    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474690    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qx4\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474732    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474765    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.474883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474844    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.475099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474888    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.475099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.474950    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.576481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576438    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qx4\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.576686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576499    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.576686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576552    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.576686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576622    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.576686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576667    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576727    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576852    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576897    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.576928    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.577228    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.577468    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.577799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.577772    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.578092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.578071    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.578303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.578285    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.580088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.580060    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.584363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.584292    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.597863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.597805    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qx4\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.598170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.598139    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\") pod \"e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") " pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:40.707156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:40.707124    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:41.069942 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:41.069906    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0fbaa_3769_406a_b110_3c9059e014d6.slice/crio-222d8eb5810bd1add608da963ce50e51a3e4cd713a25c52cfc29a5037b837dff WatchSource:0}: Error finding container 222d8eb5810bd1add608da963ce50e51a3e4cd713a25c52cfc29a5037b837dff: Status 404 returned error can't find the container with id 222d8eb5810bd1add608da963ce50e51a3e4cd713a25c52cfc29a5037b837dff
Nov 03 20:22:41.072557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.072510    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:22:41.073472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.073431    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:41.073625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.073541    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:41.073695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.073626    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:41.922624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.922585    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca816181-83ac-44a1-9ac1-617a1dfbf30b" path="/var/lib/kubelet/pods/ca816181-83ac-44a1-9ac1-617a1dfbf30b/volumes"
Nov 03 20:22:41.923504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.923251    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef33c3da-1def-4d55-b884-36cf8c93de33" path="/var/lib/kubelet/pods/ef33c3da-1def-4d55-b884-36cf8c93de33/volumes"
Nov 03 20:22:41.923744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.923717    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" event={"ID":"07b0fbaa-3769-406a-b110-3c9059e014d6","Type":"ContainerStarted","Data":"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"}
Nov 03 20:22:41.923823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.923785    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" event={"ID":"07b0fbaa-3769-406a-b110-3c9059e014d6","Type":"ContainerStarted","Data":"222d8eb5810bd1add608da963ce50e51a3e4cd713a25c52cfc29a5037b837dff"}
Nov 03 20:22:41.977195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:41.977120    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" podStartSLOduration=1.977097983 podStartE2EDuration="1.977097983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:41.973174853 +0000 UTC m=+1104.182190851" watchObservedRunningTime="2026-11-03 20:22:41.977097983 +0000 UTC m=+1104.186113981"
Nov 03 20:22:42.295686 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:42.295650    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:22:42.708113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:42.708025    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:42.714231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:42.714199    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:42.928914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:42.928877    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:42.930040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:42.930019    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:22:43.435950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:43.435915    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:22:47.756563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:47.755592    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"]
Nov 03 20:22:47.772606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:47.771799    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11a7d765-kmzhv"]
Nov 03 20:22:47.921117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:47.921073    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b917a0a-36bb-485d-a732-9fc7bc666455" path="/var/lib/kubelet/pods/9b917a0a-36bb-485d-a732-9fc7bc666455/volumes"
Nov 03 20:22:48.034306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.034228    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"]
Nov 03 20:22:48.046770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.046730    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3141c420-pr9qr"]
Nov 03 20:22:48.225878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.225844    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:22:48.226146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.226097    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" podUID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" containerName="istio-proxy" containerID="cri-o://f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a" gracePeriod=30
Nov 03 20:22:48.438850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.438599    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" podUID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" containerName="payload-pre-processing" containerID="cri-o://358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978" gracePeriod=25
Nov 03 20:22:48.714420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.714318    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"]
Nov 03 20:22:48.726542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.726491    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.728907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.728885    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"]
Nov 03 20:22:48.741434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.741407    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:22:48.741434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.741429    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:22:48.741681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.741474    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:22:48.741681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.741479    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:22:48.763829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.763802    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:22:48.796914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.796886    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzz75\" (UniqueName: \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.797060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.796997    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.797149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.797134    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.898833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.898803    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\" (UniqueName: \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\") pod \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") "
Nov 03 20:22:48.898991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.898842    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\") pod \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\" (UID: \"6c509a3c-56b3-4cc4-943b-eb013f9a794f\") "
Nov 03 20:22:48.899237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.899214    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume" pod "6c509a3c-56b3-4cc4-943b-eb013f9a794f" (UID: "6c509a3c-56b3-4cc4-943b-eb013f9a794f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:48.899330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.899315    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.899616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.899593    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz75\" (UniqueName: \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.899846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.899830    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.899983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.899971    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c509a3c-56b3-4cc4-943b-eb013f9a794f-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.900421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.900396    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.901261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.901235    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz" pod "6c509a3c-56b3-4cc4-943b-eb013f9a794f" (UID: "6c509a3c-56b3-4cc4-943b-eb013f9a794f"). InnerVolumeSpecName "kube-api-access-6sxfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:48.901771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.901752    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:48.920810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:48.920779    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz75\" (UniqueName: \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\") pod \"maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:49.011063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.010071    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxfz\" (UniqueName: \"kubernetes.io/projected/6c509a3c-56b3-4cc4-943b-eb013f9a794f-kube-api-access-6sxfz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:49.033078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.033037    5424 generic.go:356] "Generic (PLEG): container finished" podID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" containerID="358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978" exitCode=0
Nov 03 20:22:49.033352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.033292    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" event={"ID":"6c509a3c-56b3-4cc4-943b-eb013f9a794f","Type":"ContainerDied","Data":"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"}
Nov 03 20:22:49.033352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.033337    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv" event={"ID":"6c509a3c-56b3-4cc4-943b-eb013f9a794f","Type":"ContainerDied","Data":"ad2f1c0525d282bf2b1863c8026f110addcdd54fc6008ac5a4c73ea0d7befcb8"}
Nov 03 20:22:49.033454 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.033360    5424 scope.go:122] "RemoveContainer" containerID="358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"
Nov 03 20:22:49.033567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.033544    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"
Nov 03 20:22:49.050627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.050597    5424 scope.go:122] "RemoveContainer" containerID="358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"
Nov 03 20:22:49.050915 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:49.050894    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978\": container with ID starting with 358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978 not found: ID does not exist" containerID="358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"
Nov 03 20:22:49.050979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.050924    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978"} err="failed to get container status \"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978\": rpc error: code = NotFound desc = could not find container \"358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978\": container with ID starting with 358d220467d01e112a8174322ef58541066aeec532f9db3aee72572db1081978 not found: ID does not exist"
Nov 03 20:22:49.072505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.072478    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:49.090260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.090226    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:22:49.096369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.096338    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11a7d765-69f6744767-nhwkv"]
Nov 03 20:22:49.501557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.485225    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"]
Nov 03 20:22:49.925302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.925265    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" path="/var/lib/kubelet/pods/6c509a3c-56b3-4cc4-943b-eb013f9a794f/volumes"
Nov 03 20:22:49.926007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:49.925985    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d63535-f692-4a43-aa83-b7254035c360" path="/var/lib/kubelet/pods/a6d63535-f692-4a43-aa83-b7254035c360/volumes"
Nov 03 20:22:50.048502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:50.048461    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh" event={"ID":"aeb4c835-5799-4ad2-b2bd-025a1c4af467","Type":"ContainerStarted","Data":"3deb2cbe0d911b4d8b2f5d18b9c7bd0e3c0ca72682e312191d0e9dfd28059ccd"}
Nov 03 20:22:50.048737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:50.048520    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh" event={"ID":"aeb4c835-5799-4ad2-b2bd-025a1c4af467","Type":"ContainerStarted","Data":"63db1a9a5f8de36aadd1041a28f3ba7c3989379737174c305f3e83890d5ff469"}
Nov 03 20:22:50.091605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:50.091543    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh" podStartSLOduration=2.091512475 podStartE2EDuration="2.091512475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:50.085727569 +0000 UTC m=+1112.294743569" watchObservedRunningTime="2026-11-03 20:22:50.091512475 +0000 UTC m=+1112.300528474"
Nov 03 20:22:51.034585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:51.032306    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"]
Nov 03 20:22:51.051622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:51.047015    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03fd5fb-p87cv"]
Nov 03 20:22:51.069384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:51.069343    5424 generic.go:356] "Generic (PLEG): container finished" podID="aeb4c835-5799-4ad2-b2bd-025a1c4af467" containerID="3deb2cbe0d911b4d8b2f5d18b9c7bd0e3c0ca72682e312191d0e9dfd28059ccd" exitCode=0
Nov 03 20:22:51.069737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:51.069699    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh" event={"ID":"aeb4c835-5799-4ad2-b2bd-025a1c4af467","Type":"ContainerDied","Data":"3deb2cbe0d911b4d8b2f5d18b9c7bd0e3c0ca72682e312191d0e9dfd28059ccd"}
Nov 03 20:22:51.925981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:51.925946    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2948a86-7ec2-4a58-9271-65de8f119b37" path="/var/lib/kubelet/pods/a2948a86-7ec2-4a58-9271-65de8f119b37/volumes"
Nov 03 20:22:52.246565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.246520    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:52.287643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.287610    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\") pod \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") "
Nov 03 20:22:52.287874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.287666    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\") pod \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") "
Nov 03 20:22:52.287874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.287810    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\" (UniqueName: \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\") pod \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\" (UID: \"aeb4c835-5799-4ad2-b2bd-025a1c4af467\") "
Nov 03 20:22:52.288054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.288028    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca" pod "aeb4c835-5799-4ad2-b2bd-025a1c4af467" (UID: "aeb4c835-5799-4ad2-b2bd-025a1c4af467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:52.288571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.288546    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:52.291097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.291067    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca" pod "aeb4c835-5799-4ad2-b2bd-025a1c4af467" (UID: "aeb4c835-5799-4ad2-b2bd-025a1c4af467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:52.295173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.295143    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75" pod "aeb4c835-5799-4ad2-b2bd-025a1c4af467" (UID: "aeb4c835-5799-4ad2-b2bd-025a1c4af467"). InnerVolumeSpecName "kube-api-access-vzz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:52.389516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.389483    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4c835-5799-4ad2-b2bd-025a1c4af467-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:52.389516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:52.389514    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzz75\" (UniqueName: \"kubernetes.io/projected/aeb4c835-5799-4ad2-b2bd-025a1c4af467-kube-api-access-vzz75\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.087386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.087353    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"
Nov 03 20:22:53.087628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.087397    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh" event={"ID":"aeb4c835-5799-4ad2-b2bd-025a1c4af467","Type":"ContainerDied","Data":"63db1a9a5f8de36aadd1041a28f3ba7c3989379737174c305f3e83890d5ff469"}
Nov 03 20:22:53.087628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.087453    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63db1a9a5f8de36aadd1041a28f3ba7c3989379737174c305f3e83890d5ff469"
Nov 03 20:22:53.546922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.546895    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:22:53.606473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606443    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.606656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606613    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.606755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606741    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.606843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606773    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.606843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606834    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.606952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606900    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.607007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.606984    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.607064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607023    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.607143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607129    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\") pod \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\" (UID: \"33d5bb95-f63d-44ba-bfec-34481b8f12e0\") "
Nov 03 20:22:53.607283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607255    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:53.607823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607583    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:53.607823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607776    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:53.607823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607793    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:53.607823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607807    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:53.608146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607943    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.608146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607963    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.608146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607977    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.608146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.607989    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.608146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.608001    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.609961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.609939    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "kube-api-access-nzn29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:53.610627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.610598    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:53.611446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.611420    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:53.611808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.611783    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token" pod "33d5bb95-f63d-44ba-bfec-34481b8f12e0" (UID: "33d5bb95-f63d-44ba-bfec-34481b8f12e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:53.709945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.709909    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.709945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.709941    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.710146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.709953    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzn29\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-kube-api-access-nzn29\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:53.710146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:53.709967    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d5bb95-f63d-44ba-bfec-34481b8f12e0-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:22:54.097820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.097786    5424 generic.go:356] "Generic (PLEG): container finished" podID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" containerID="f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a" exitCode=0
Nov 03 20:22:54.097995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.097834    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"
Nov 03 20:22:54.097995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.097872    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" event={"ID":"33d5bb95-f63d-44ba-bfec-34481b8f12e0","Type":"ContainerDied","Data":"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"}
Nov 03 20:22:54.097995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.097939    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f" event={"ID":"33d5bb95-f63d-44ba-bfec-34481b8f12e0","Type":"ContainerDied","Data":"677e3d8665e55fae7948d50094c991d76215706dae6661a950083504b981db8b"}
Nov 03 20:22:54.097995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.097967    5424 scope.go:122] "RemoveContainer" containerID="f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"
Nov 03 20:22:54.110207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.110184    5424 scope.go:122] "RemoveContainer" containerID="f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"
Nov 03 20:22:54.110558 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:22:54.110505    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a\": container with ID starting with f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a not found: ID does not exist" containerID="f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"
Nov 03 20:22:54.110647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.110550    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a"} err="failed to get container status \"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a\": rpc error: code = NotFound desc = could not find container \"f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a\": container with ID starting with f6476c7f4b59f594cbcc8904f12121da5f9a86dd220c0ee1ad2f2cf83515f17a not found: ID does not exist"
Nov 03 20:22:54.164825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.164791    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:22:54.176707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.176677    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-11a7d765-openshift-default-775d47679c-6d28f"]
Nov 03 20:22:54.578929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.578891    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:22:54.579589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579572    5424 cpu_manager.go:414] "containerMap: removing container" podUID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" containerName="payload-pre-processing"
Nov 03 20:22:54.579657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579589    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" containerName="payload-pre-processing"
Nov 03 20:22:54.579657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579602    5424 cpu_manager.go:414] "containerMap: removing container" podUID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" containerName="istio-proxy"
Nov 03 20:22:54.579657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579606    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" containerName="istio-proxy"
Nov 03 20:22:54.579657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579616    5424 cpu_manager.go:414] "containerMap: removing container" podUID="aeb4c835-5799-4ad2-b2bd-025a1c4af467" containerName="revoke-keys"
Nov 03 20:22:54.579657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579621    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb4c835-5799-4ad2-b2bd-025a1c4af467" containerName="revoke-keys"
Nov 03 20:22:54.579885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579721    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c509a3c-56b3-4cc4-943b-eb013f9a794f" containerName="payload-pre-processing"
Nov 03 20:22:54.579885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579728    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" containerName="istio-proxy"
Nov 03 20:22:54.579885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.579734    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb4c835-5799-4ad2-b2bd-025a1c4af467" containerName="revoke-keys"
Nov 03 20:22:54.585304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.585279    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.600882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.600849    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a619f336-dockercfg-pwqxs\""
Nov 03 20:22:54.600882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.600852    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a619f336\""
Nov 03 20:22:54.603586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.603560    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:22:54.625578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.625469    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sk86\" (UniqueName: \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.625759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.625645    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.634767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.634733    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:22:54.640479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.640457    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.658123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.658088    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:22:54.727017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.726977    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.727218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.727138    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.727218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.727188    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sk86\" (UniqueName: \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.727327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.727304    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.728238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.728215    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.746659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.746631    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sk86\" (UniqueName: \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\") pod \"payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") " pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.831101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.830941    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.831428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.831404    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.833491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.833469    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.900973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.899615    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\") pod \"payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:54.905278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.905030    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:54.977669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:54.969835    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:55.060641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.060605    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"]
Nov 03 20:22:55.136695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.136281    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81b74f1e-thzpp"]
Nov 03 20:22:55.460971 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:22:55.459429    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f91250_9849_4f75_82b2_833a1b476bf0.slice/crio-76fdb4d0f680285feda03528cb3c7b0c1eece868e9e9fc2e18561e4d8296aa5e WatchSource:0}: Error finding container 76fdb4d0f680285feda03528cb3c7b0c1eece868e9e9fc2e18561e4d8296aa5e: Status 404 returned error can't find the container with id 76fdb4d0f680285feda03528cb3c7b0c1eece868e9e9fc2e18561e4d8296aa5e
Nov 03 20:22:55.467256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.467209    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:22:55.523198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.523143    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:22:55.935557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.926593    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d5bb95-f63d-44ba-bfec-34481b8f12e0" path="/var/lib/kubelet/pods/33d5bb95-f63d-44ba-bfec-34481b8f12e0/volumes"
Nov 03 20:22:55.935557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:55.927643    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf9f8e1-02d2-4000-8724-b5a3dc9adb41" path="/var/lib/kubelet/pods/3cf9f8e1-02d2-4000-8724-b5a3dc9adb41/volumes"
Nov 03 20:22:56.161638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.161585    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" event={"ID":"11f91250-9849-4f75-82b2-833a1b476bf0","Type":"ContainerStarted","Data":"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"}
Nov 03 20:22:56.161866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.161671    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" event={"ID":"11f91250-9849-4f75-82b2-833a1b476bf0","Type":"ContainerStarted","Data":"76fdb4d0f680285feda03528cb3c7b0c1eece868e9e9fc2e18561e4d8296aa5e"}
Nov 03 20:22:56.165566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.162941    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:22:56.175561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.169111    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" event={"ID":"acb83283-8519-47ab-b5c9-c213b417d726","Type":"ContainerStarted","Data":"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"}
Nov 03 20:22:56.175561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.169173    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" event={"ID":"acb83283-8519-47ab-b5c9-c213b417d726","Type":"ContainerStarted","Data":"1c65a4829a85219c5ec8ead907cb411fe2b01ad7424fd1e137917bb73b6aaab1"}
Nov 03 20:22:56.175561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.170142    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:22:56.210641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:56.210397    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" podStartSLOduration=2.210374069 podStartE2EDuration="2.210374069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:56.20278085 +0000 UTC m=+1118.411796871" watchObservedRunningTime="2026-11-03 20:22:56.210374069 +0000 UTC m=+1118.419390066"
Nov 03 20:22:57.366553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.366430    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:22:57.465556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.464652    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:57.502559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.490031    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:57.603556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.602972    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:22:57.616666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.614645    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:57.632554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.618688    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:57.721555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.707616    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:22:57.856555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.856059    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:22:57.986614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.968616    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:58.007584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:57.991645    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:58.385914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:58.383923    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:58.385914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:58.384108    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:22:58.396873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:58.396781    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:22:58.481557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:58.481312    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" podStartSLOduration=4.481292184 podStartE2EDuration="4.481292184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:56.25446648 +0000 UTC m=+1118.463482515" watchObservedRunningTime="2026-11-03 20:22:58.481292184 +0000 UTC m=+1120.690308182"
Nov 03 20:22:59.347391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:22:59.347349    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:23:00.059244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:00.059180    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"]
Nov 03 20:23:00.068277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:00.068231    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89e30d0a-wwmw6"]
Nov 03 20:23:01.921386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:01.921314    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6f0f5d-92c0-49ac-bd71-22251da42cea" path="/var/lib/kubelet/pods/3a6f0f5d-92c0-49ac-bd71-22251da42cea/volumes"
Nov 03 20:23:03.030257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:03.030220    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"]
Nov 03 20:23:03.044222 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:03.044183    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-024356e0-8phw8"]
Nov 03 20:23:03.923362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:03.923323    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbe545c-d818-482e-ba4e-39347358dfb7" path="/var/lib/kubelet/pods/0bbe545c-d818-482e-ba4e-39347358dfb7/volumes"
Nov 03 20:23:04.348066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.348027    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" podUID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" containerName="payload-pre-processing" containerID="cri-o://10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d" gracePeriod=25
Nov 03 20:23:04.393346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.393315    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"]
Nov 03 20:23:04.406987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.406956    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b67df53-zfvzh"]
Nov 03 20:23:04.519616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.519576    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:23:04.521482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.520151    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerName="istio-proxy" containerID="cri-o://5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905" gracePeriod=30
Nov 03 20:23:04.759099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.759067    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:23:04.927215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.926984    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\" (UniqueName: \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\") pod \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") "
Nov 03 20:23:04.927405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.927353    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\") pod \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\" (UID: \"5b18f66c-56ea-4954-9bdb-6562bd36afd4\") "
Nov 03 20:23:04.927834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.927801    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume" pod "5b18f66c-56ea-4954-9bdb-6562bd36afd4" (UID: "5b18f66c-56ea-4954-9bdb-6562bd36afd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:04.928057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.928030    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b18f66c-56ea-4954-9bdb-6562bd36afd4-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:04.933600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:04.931694    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m" pod "5b18f66c-56ea-4954-9bdb-6562bd36afd4" (UID: "5b18f66c-56ea-4954-9bdb-6562bd36afd4"). InnerVolumeSpecName "kube-api-access-79z8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.029904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.029872    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79z8m\" (UniqueName: \"kubernetes.io/projected/5b18f66c-56ea-4954-9bdb-6562bd36afd4-kube-api-access-79z8m\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:05.032837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.032807    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"]
Nov 03 20:23:05.039595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.038008    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47e4d6d-tb4t4"]
Nov 03 20:23:05.046644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.046615    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"]
Nov 03 20:23:05.048118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.048099    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7735e22c-bcqvv"]
Nov 03 20:23:05.350997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.350905    5424 generic.go:356] "Generic (PLEG): container finished" podID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" containerID="10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d" exitCode=0
Nov 03 20:23:05.351602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.350998    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"
Nov 03 20:23:05.351602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.350993    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" event={"ID":"5b18f66c-56ea-4954-9bdb-6562bd36afd4","Type":"ContainerDied","Data":"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"}
Nov 03 20:23:05.351602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.351144    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl" event={"ID":"5b18f66c-56ea-4954-9bdb-6562bd36afd4","Type":"ContainerDied","Data":"75858aa034ccd1db1d80fb8653c977a6add2a2a1e231cbbbfb61375b7b53d312"}
Nov 03 20:23:05.351602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.351178    5424 scope.go:122] "RemoveContainer" containerID="10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"
Nov 03 20:23:05.365330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.365291    5424 scope.go:122] "RemoveContainer" containerID="10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"
Nov 03 20:23:05.365680 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:05.365661    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d\": container with ID starting with 10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d not found: ID does not exist" containerID="10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"
Nov 03 20:23:05.365769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.365689    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d"} err="failed to get container status \"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d\": rpc error: code = NotFound desc = could not find container \"10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d\": container with ID starting with 10c28b739321e8e4433b7e0d03d5efd3a058be0903a586d80ff8b3bb8a80621d not found: ID does not exist"
Nov 03 20:23:05.429520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.429489    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:23:05.451981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.451956    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b67df53-7998994879-nhkwl"]
Nov 03 20:23:05.921934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.921901    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fec1ace-e27f-4573-9e04-cdf1dbc79ac4" path="/var/lib/kubelet/pods/2fec1ace-e27f-4573-9e04-cdf1dbc79ac4/volumes"
Nov 03 20:23:05.922446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.922424    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" path="/var/lib/kubelet/pods/5b18f66c-56ea-4954-9bdb-6562bd36afd4/volumes"
Nov 03 20:23:05.922971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.922949    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d870b6e-91e5-4348-963a-15bc827e318f" path="/var/lib/kubelet/pods/7d870b6e-91e5-4348-963a-15bc827e318f/volumes"
Nov 03 20:23:05.923783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:05.923762    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb4c835-5799-4ad2-b2bd-025a1c4af467" path="/var/lib/kubelet/pods/aeb4c835-5799-4ad2-b2bd-025a1c4af467/volumes"
Nov 03 20:23:06.015803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.015752    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:23:06.145874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.145837    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.145916    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.145953    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146041    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146096    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146167    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146197    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:06.146381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146260    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146352    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146389    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:06.146487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146397    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:06.146487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146471    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\") pod \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\" (UID: \"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0\") "
Nov 03 20:23:06.146664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146566    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:06.146664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.146617    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:06.147227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.147203    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.147227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.147228    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.147613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.147241    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.147613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.147249    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.147613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.147261    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.149152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.149120    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "kube-api-access-rktn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:06.149278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.149168    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:06.149358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.149301    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:06.149358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.149317    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token" pod "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" (UID: "1d978e65-5004-4abe-b2d7-21c2e2e0e1a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:06.177609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.177581    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:23:06.178255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178233    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" containerName="payload-pre-processing"
Nov 03 20:23:06.178360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178256    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" containerName="payload-pre-processing"
Nov 03 20:23:06.178360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178274    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerName="istio-proxy"
Nov 03 20:23:06.178360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178281    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerName="istio-proxy"
Nov 03 20:23:06.178518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178441    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b18f66c-56ea-4954-9bdb-6562bd36afd4" containerName="payload-pre-processing"
Nov 03 20:23:06.178518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.178456    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerName="istio-proxy"
Nov 03 20:23:06.181823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.181803    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.200023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.199970    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b917c2-openshift-default-dockercfg-ldfmq\""
Nov 03 20:23:06.223449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.223417    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:23:06.248710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.248682    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.248710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.248708    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.248897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.248718    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.248897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.248729    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rktn7\" (UniqueName: \"kubernetes.io/projected/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0-kube-api-access-rktn7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:06.349662 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.349567    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.349662 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.349640    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhw9\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.349962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.349826    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.349962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.349896    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.350035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.349996    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.350241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.350212    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.350299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.350262    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.350386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.350366    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.350438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.350424    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.362322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.362287    5424 generic.go:356] "Generic (PLEG): container finished" podID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" containerID="5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905" exitCode=0
Nov 03 20:23:06.362731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.362636    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" event={"ID":"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0","Type":"ContainerDied","Data":"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"}
Nov 03 20:23:06.362731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.362694    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb" event={"ID":"1d978e65-5004-4abe-b2d7-21c2e2e0e1a0","Type":"ContainerDied","Data":"b8166774703f221f172593e7558108171252b7ca8620f6cc353271454c540094"}
Nov 03 20:23:06.362731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.362727    5424 scope.go:122] "RemoveContainer" containerID="5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"
Nov 03 20:23:06.362874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.362859    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"
Nov 03 20:23:06.377643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.377619    5424 scope.go:122] "RemoveContainer" containerID="5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"
Nov 03 20:23:06.377954 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:06.377929    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905\": container with ID starting with 5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905 not found: ID does not exist" containerID="5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"
Nov 03 20:23:06.378067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.377961    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905"} err="failed to get container status \"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905\": rpc error: code = NotFound desc = could not find container \"5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905\": container with ID starting with 5ae18e317a2542a4e056652aa475839353d532c6b7c774ea958e6a123bb33905 not found: ID does not exist"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.464943    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465035    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465079    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465228    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465260    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465311    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465345    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465420    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.465478    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhw9\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.466631    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.467165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.467567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.467372    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.481941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.481890    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.482221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.482133    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.493573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.490353    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.518555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.496586    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:23:06.519248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.519181    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.526577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.523976    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.541574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.532121    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhw9\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\") pod \"e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") " pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:06.545245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.545211    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b67df53-openshift-default-65577cc6d9-5l5fb"]
Nov 03 20:23:06.797593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:06.797557    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:07.142495 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:07.142460    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018e100d_188f_4968_ad37_869b18b33f59.slice/crio-6d0ab02b5874f93ca1d0005a6ed23fa5b7f3f1c7932675c4c0bcc07de88bb84c WatchSource:0}: Error finding container 6d0ab02b5874f93ca1d0005a6ed23fa5b7f3f1c7932675c4c0bcc07de88bb84c: Status 404 returned error can't find the container with id 6d0ab02b5874f93ca1d0005a6ed23fa5b7f3f1c7932675c4c0bcc07de88bb84c
Nov 03 20:23:07.144853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.144827    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:23:07.146304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.146254    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:07.146652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.146615    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:07.146854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.146828    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:07.379969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.379931    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerStarted","Data":"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"}
Nov 03 20:23:07.380426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.380023    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerStarted","Data":"6d0ab02b5874f93ca1d0005a6ed23fa5b7f3f1c7932675c4c0bcc07de88bb84c"}
Nov 03 20:23:07.428696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.428588    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podStartSLOduration=1.428567123 podStartE2EDuration="1.428567123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:07.426887489 +0000 UTC m=+1129.635903513" watchObservedRunningTime="2026-11-03 20:23:07.428567123 +0000 UTC m=+1129.637583118"
Nov 03 20:23:07.798692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.798659    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:07.921987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:07.921951    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d978e65-5004-4abe-b2d7-21c2e2e0e1a0" path="/var/lib/kubelet/pods/1d978e65-5004-4abe-b2d7-21c2e2e0e1a0/volumes"
Nov 03 20:23:08.197095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.196980    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:23:08.201755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.201718    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.231328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.230015    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:23:08.231328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.230124    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gwj92\""
Nov 03 20:23:08.231328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.230180    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:23:08.231328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.230225    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:23:08.231697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.231384    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:23:08.260065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.260038    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:23:08.260382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.260355    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:23:08.302160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302119    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302355    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302453    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302496    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302564    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8kv\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302648    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.303406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.302814    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413340    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413408    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413462    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8kv\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413564    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413723    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.413881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413758    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.414130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.413933    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.415851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.415825    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.421456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.421394    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.422123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.422100    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.423175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.423116    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.430270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.430246    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.434122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.434101    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8kv\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.438124 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.438105    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r76c7\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.528104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.528068    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:08.821180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.821087    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:08.858541 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:08.858501    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ef0547_476a_4b95_a7ea_8d38681afc88.slice/crio-519664b9d7ba8d34b26148ecd4df394b16257884a1331f67d84eaa4153b58111 WatchSource:0}: Error finding container 519664b9d7ba8d34b26148ecd4df394b16257884a1331f67d84eaa4153b58111: Status 404 returned error can't find the container with id 519664b9d7ba8d34b26148ecd4df394b16257884a1331f67d84eaa4153b58111
Nov 03 20:23:08.859052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:08.859031    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:23:09.408108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:09.408076    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" event={"ID":"e4ef0547-476a-4b95-a7ea-8d38681afc88","Type":"ContainerStarted","Data":"519664b9d7ba8d34b26148ecd4df394b16257884a1331f67d84eaa4153b58111"}
Nov 03 20:23:09.408343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:09.408324    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:09.409316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:09.409298    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:23:11.954613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:11.954580    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:11.954963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:11.954642    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:12.295577 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:12.292062    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:23:12.533953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:12.531626    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" event={"ID":"e4ef0547-476a-4b95-a7ea-8d38681afc88","Type":"ContainerStarted","Data":"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"}
Nov 03 20:23:12.533953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:12.531903    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:13.560207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:13.560173    5424 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r76c7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:23:13.560738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:13.560229    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:23:15.029184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:15.029130    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" podStartSLOduration=3.937105056 podStartE2EDuration="7.029115621s" totalImagesPullingTime="3.092010565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:12.597993435 +0000 UTC m=+1134.807009433" watchObservedRunningTime="2026-11-03 20:23:15.029115621 +0000 UTC m=+1137.238131617"
Nov 03 20:23:15.029729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:15.029713    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"]
Nov 03 20:23:15.041601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:15.041576    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd4cce80-wpzvp"]
Nov 03 20:23:15.921273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:15.921242    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c8452f-59d8-43b7-a1c6-2c56e398e105" path="/var/lib/kubelet/pods/44c8452f-59d8-43b7-a1c6-2c56e398e105/volumes"
Nov 03 20:23:16.559476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:16.559448    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:23:17.032511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:17.032469    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"]
Nov 03 20:23:17.037965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:17.037934    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7144e1e8-2vmm9"]
Nov 03 20:23:17.928323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:17.928290    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74526ef8-b63b-45da-b6b7-c1bdbf28e8b9" path="/var/lib/kubelet/pods/74526ef8-b63b-45da-b6b7-c1bdbf28e8b9/volumes"
Nov 03 20:23:18.034432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:18.034399    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"]
Nov 03 20:23:18.046464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:18.046443    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a740d1b5-mkjs2"]
Nov 03 20:23:19.922726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:19.922686    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337f958e-4d25-452f-97f2-230370c2f6bd" path="/var/lib/kubelet/pods/337f958e-4d25-452f-97f2-230370c2f6bd/volumes"
Nov 03 20:23:25.090776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.090735    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"]
Nov 03 20:23:25.101190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.101171    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.113759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.113734    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"]
Nov 03 20:23:25.119374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.119342    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:23:25.119615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.119593    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:25.119752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.119643    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:23:25.128148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.128127    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:23:25.161939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.161915    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.162042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.161961    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.162042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.162013    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlg2\" (UniqueName: \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.263047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.263019    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.263225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.263071    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.263225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.263099    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlg2\" (UniqueName: \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.263832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.263803    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.265582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.265564    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.283023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.282995    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlg2\" (UniqueName: \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\") pod \"maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.422732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.422657    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:25.730666 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:25.730626    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e4b890_e592_4461_a7a7_e1e7b2a6e22d.slice/crio-fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef WatchSource:0}: Error finding container fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef: Status 404 returned error can't find the container with id fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef
Nov 03 20:23:25.733383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:25.733359    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"]
Nov 03 20:23:26.689591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:26.687708    5424 generic.go:356] "Generic (PLEG): container finished" podID="e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" containerID="2497fbf248ec83b0681f2cb6979b7e6e46ffcba19aef5fcdae31d82020880e6b" exitCode=0
Nov 03 20:23:26.689591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:26.687899    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz" event={"ID":"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d","Type":"ContainerDied","Data":"2497fbf248ec83b0681f2cb6979b7e6e46ffcba19aef5fcdae31d82020880e6b"}
Nov 03 20:23:26.689591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:26.687954    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz" event={"ID":"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d","Type":"ContainerStarted","Data":"fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef"}
Nov 03 20:23:27.033000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:27.032968    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"]
Nov 03 20:23:27.035286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:27.035256    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6544f52d-twf5p"]
Nov 03 20:23:27.953574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:27.953496    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a" path="/var/lib/kubelet/pods/cd6acbd0-a1ba-43de-b89a-9cc3e8431f4a/volumes"
Nov 03 20:23:27.961977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:27.961954    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:28.021263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.021237    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\") pod \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") "
Nov 03 20:23:28.021457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.021335    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\") pod \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") "
Nov 03 20:23:28.021522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.021467    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\" (UniqueName: \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\") pod \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\" (UID: \"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d\") "
Nov 03 20:23:28.022045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.022016    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca" pod "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" (UID: "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:28.024568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.024498    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:28.029931 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.029904    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2" pod "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" (UID: "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d"). InnerVolumeSpecName "kube-api-access-gqlg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:28.031775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.031751    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca" pod "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" (UID: "e5e4b890-e592-4461-a7a7-e1e7b2a6e22d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:28.126320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.126290    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:28.126320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.126318    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlg2\" (UniqueName: \"kubernetes.io/projected/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d-kube-api-access-gqlg2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:28.720768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.720726    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz" event={"ID":"e5e4b890-e592-4461-a7a7-e1e7b2a6e22d","Type":"ContainerDied","Data":"fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef"}
Nov 03 20:23:28.720768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.720762    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"
Nov 03 20:23:28.721061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:28.720790    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe59a38d4ff5a98d40be84fef5a591347a57c262dc5ecb426fffe1d878ef41ef"
Nov 03 20:23:32.908117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.908081    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"]
Nov 03 20:23:32.908734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.908718    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" containerName="revoke-keys"
Nov 03 20:23:32.908797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.908736    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" containerName="revoke-keys"
Nov 03 20:23:32.908870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.908861    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" containerName="revoke-keys"
Nov 03 20:23:32.912921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.912893    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:32.954277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.954241    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"]
Nov 03 20:23:32.960323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.960292    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:23:32.960634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.960610    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:32.960780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.960622    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:23:32.960780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.960698    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:23:32.977346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.977320    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:32.977451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.977392    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:32.977587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:32.977562    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglbz\" (UniqueName: \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.079160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.079126    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.079374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.079223    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.079374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.079292    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglbz\" (UniqueName: \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.079914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.079890    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.081772 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.081752    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.098639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.098614    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglbz\" (UniqueName: \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\") pod \"maas-api-revoke-keys-e2e-derive-a619f336-xtt4s\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.262018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.261980    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:33.647268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.647240    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"]
Nov 03 20:23:33.780700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:33.780660    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s" event={"ID":"0460dfef-66f0-45d3-91ff-d3a5403e4e77","Type":"ContainerStarted","Data":"ff1088fb679d4d9956bcd44e05b093d1f19bceda9aac8b7b6e33348be25050d5"}
Nov 03 20:23:34.790439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:34.790407    5424 generic.go:356] "Generic (PLEG): container finished" podID="0460dfef-66f0-45d3-91ff-d3a5403e4e77" containerID="4ed88b39b7c33babd996028d17ff093e43b600fb5378e53e260fede058222de7" exitCode=0
Nov 03 20:23:34.790776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:34.790498    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s" event={"ID":"0460dfef-66f0-45d3-91ff-d3a5403e4e77","Type":"ContainerDied","Data":"4ed88b39b7c33babd996028d17ff093e43b600fb5378e53e260fede058222de7"}
Nov 03 20:23:36.267613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.267580    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:36.380609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.380571    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\" (UniqueName: \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\") pod \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") "
Nov 03 20:23:36.381291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.381263    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\") pod \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") "
Nov 03 20:23:36.381740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.381723    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\") pod \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\" (UID: \"0460dfef-66f0-45d3-91ff-d3a5403e4e77\") "
Nov 03 20:23:36.382994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.382976    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca" pod "0460dfef-66f0-45d3-91ff-d3a5403e4e77" (UID: "0460dfef-66f0-45d3-91ff-d3a5403e4e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:36.387919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.387898    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca" pod "0460dfef-66f0-45d3-91ff-d3a5403e4e77" (UID: "0460dfef-66f0-45d3-91ff-d3a5403e4e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:36.392016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.391993    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz" pod "0460dfef-66f0-45d3-91ff-d3a5403e4e77" (UID: "0460dfef-66f0-45d3-91ff-d3a5403e4e77"). InnerVolumeSpecName "kube-api-access-pglbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:36.498145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.498104    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:36.498368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.498351    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglbz\" (UniqueName: \"kubernetes.io/projected/0460dfef-66f0-45d3-91ff-d3a5403e4e77-kube-api-access-pglbz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:36.501561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.499614    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0460dfef-66f0-45d3-91ff-d3a5403e4e77-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:36.843290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.843108    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"
Nov 03 20:23:36.843469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.843334    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s" event={"ID":"0460dfef-66f0-45d3-91ff-d3a5403e4e77","Type":"ContainerDied","Data":"ff1088fb679d4d9956bcd44e05b093d1f19bceda9aac8b7b6e33348be25050d5"}
Nov 03 20:23:36.843469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:36.843379    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1088fb679d4d9956bcd44e05b093d1f19bceda9aac8b7b6e33348be25050d5"
Nov 03 20:23:38.144134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:38.144100    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"]
Nov 03 20:23:38.149727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:38.149691    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04bf5fc9-7jktz"]
Nov 03 20:23:39.579240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:39.579204    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:23:39.580085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:39.580040    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" podUID="07b0fbaa-3769-406a-b110-3c9059e014d6" containerName="istio-proxy" containerID="cri-o://fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5" gracePeriod=30
Nov 03 20:23:39.949029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:39.948944    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e4b890-e592-4461-a7a7-e1e7b2a6e22d" path="/var/lib/kubelet/pods/e5e4b890-e592-4461-a7a7-e1e7b2a6e22d/volumes"
Nov 03 20:23:40.707460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.707430    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:23:40.708268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.708246    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0460dfef-66f0-45d3-91ff-d3a5403e4e77" containerName="revoke-keys"
Nov 03 20:23:40.708268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.708269    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0460dfef-66f0-45d3-91ff-d3a5403e4e77" containerName="revoke-keys"
Nov 03 20:23:40.708478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.708459    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0460dfef-66f0-45d3-91ff-d3a5403e4e77" containerName="revoke-keys"
Nov 03 20:23:40.713373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.713350    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.729023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.728991    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-de82c8-openshift-default-dockercfg-9882k\""
Nov 03 20:23:40.749212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.749181    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:23:40.869172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869140    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvs4q\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869271    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869308    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869348    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869555    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869610    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869729    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869840    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.869970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.869879    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.974052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.973940    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.974330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.974306    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.974474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.974460    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvs4q\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.974708 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.974691    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.974823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.974810    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.975008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.974946    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.975258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.975237    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.975420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.975403    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.975638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.975620    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.978613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.978583    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.979244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.979195    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.979872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.979846    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.981608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.981569    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.981877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.981853    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:40.984786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:40.984725    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:41.021555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.003252    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:41.021555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.014299    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:41.021555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.014788    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvs4q\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\") pod \"e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") " pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:41.041565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.039242    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:41.440625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.440511    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:23:41.441403 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:41.441376    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff940dd_11f0_44c2_8c65_51041e56fcd7.slice/crio-c384c79588d3fbc6b40e2a6848f9deb6ea48fdc3a9db9ac23385afcf2c6e4b9b WatchSource:0}: Error finding container c384c79588d3fbc6b40e2a6848f9deb6ea48fdc3a9db9ac23385afcf2c6e4b9b: Status 404 returned error can't find the container with id c384c79588d3fbc6b40e2a6848f9deb6ea48fdc3a9db9ac23385afcf2c6e4b9b
Nov 03 20:23:41.444444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.444406    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.444562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.444495    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.444639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.444557    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.952208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.952171    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerStarted","Data":"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"}
Nov 03 20:23:41.952663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:41.952227    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerStarted","Data":"c384c79588d3fbc6b40e2a6848f9deb6ea48fdc3a9db9ac23385afcf2c6e4b9b"}
Nov 03 20:23:42.020983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.020927    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podStartSLOduration=2.020912903 podStartE2EDuration="2.020912903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:42.008035171 +0000 UTC m=+1164.217051181" watchObservedRunningTime="2026-11-03 20:23:42.020912903 +0000 UTC m=+1164.229928899"
Nov 03 20:23:42.041811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.041778    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:42.054582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.052830    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:42.304692 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:42.304657    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:23:42.700991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.700961    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:23:42.831351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831256    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831396    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831494    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831564    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:42.831675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831617    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831670    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831728    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831732    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:42.831822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831806    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831886    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.831973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831917    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\") pod \"07b0fbaa-3769-406a-b110-3c9059e014d6\" (UID: \"07b0fbaa-3769-406a-b110-3c9059e014d6\") "
Nov 03 20:23:42.832068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.831965    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:42.832229 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832195    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:42.832316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832245    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:42.832927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832859    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.832927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832887    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.832927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832901    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.832927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832917    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b0fbaa-3769-406a-b110-3c9059e014d6-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.832927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.832929    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.834811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.834781    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:42.834984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.834959    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:42.835356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.835335    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "kube-api-access-q7qx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:42.835615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.835595    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token" pod "07b0fbaa-3769-406a-b110-3c9059e014d6" (UID: "07b0fbaa-3769-406a-b110-3c9059e014d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:42.934190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.934152    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.934190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.934184    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7qx4\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-kube-api-access-q7qx4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.934190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.934194    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.934433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.934203    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b0fbaa-3769-406a-b110-3c9059e014d6-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:42.961636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.961603    5424 generic.go:356] "Generic (PLEG): container finished" podID="07b0fbaa-3769-406a-b110-3c9059e014d6" containerID="fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5" exitCode=0
Nov 03 20:23:42.962150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.961657    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"
Nov 03 20:23:42.962150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.961684    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" event={"ID":"07b0fbaa-3769-406a-b110-3c9059e014d6","Type":"ContainerDied","Data":"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"}
Nov 03 20:23:42.962150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.961769    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6" event={"ID":"07b0fbaa-3769-406a-b110-3c9059e014d6","Type":"ContainerDied","Data":"222d8eb5810bd1add608da963ce50e51a3e4cd713a25c52cfc29a5037b837dff"}
Nov 03 20:23:42.962150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.961799    5424 scope.go:122] "RemoveContainer" containerID="fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"
Nov 03 20:23:42.962150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.962112    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:42.963304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.963276    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:23:42.973849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.973829    5424 scope.go:122] "RemoveContainer" containerID="fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"
Nov 03 20:23:42.974162 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:42.974142    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5\": container with ID starting with fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5 not found: ID does not exist" containerID="fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"
Nov 03 20:23:42.974211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:42.974170    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5"} err="failed to get container status \"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5\": rpc error: code = NotFound desc = could not find container \"fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5\": container with ID starting with fc344e6dde54739f027e596ffda4b85f021923355e58f03252d189a3fbd3d6d5 not found: ID does not exist"
Nov 03 20:23:43.015800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:43.015762    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:23:43.021718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:43.021674    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-04bf5fc9-openshift-default-58cb479fbc-w2zb6"]
Nov 03 20:23:43.352233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:43.352198    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:23:43.398519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:43.398485    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:23:43.921416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:43.921385    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b0fbaa-3769-406a-b110-3c9059e014d6" path="/var/lib/kubelet/pods/07b0fbaa-3769-406a-b110-3c9059e014d6/volumes"
Nov 03 20:23:47.643104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:47.643065    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"]
Nov 03 20:23:47.658329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:47.658295    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a619f336-xtt4s"]
Nov 03 20:23:47.929403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:47.929327    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0460dfef-66f0-45d3-91ff-d3a5403e4e77" path="/var/lib/kubelet/pods/0460dfef-66f0-45d3-91ff-d3a5403e4e77/volumes"
Nov 03 20:23:48.352766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.352723    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" podUID="acb83283-8519-47ab-b5c9-c213b417d726" containerName="payload-processing" containerID="cri-o://b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181" gracePeriod=25
Nov 03 20:23:48.399546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.399490    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" podUID="11f91250-9849-4f75-82b2-833a1b476bf0" containerName="payload-pre-processing" containerID="cri-o://f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6" gracePeriod=25
Nov 03 20:23:48.440954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.440922    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"]
Nov 03 20:23:48.441933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.441908    5424 cpu_manager.go:414] "containerMap: removing container" podUID="07b0fbaa-3769-406a-b110-3c9059e014d6" containerName="istio-proxy"
Nov 03 20:23:48.442102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.442075    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b0fbaa-3769-406a-b110-3c9059e014d6" containerName="istio-proxy"
Nov 03 20:23:48.442271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.442258    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b0fbaa-3769-406a-b110-3c9059e014d6" containerName="istio-proxy"
Nov 03 20:23:48.446062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.446034    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.469781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.469729    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:23:48.469989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.469885    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:23:48.477673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.473796    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:23:48.477673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.473870    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:48.490640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.490609    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"]
Nov 03 20:23:48.544095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.543834    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.544095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.543896    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdj2\" (UniqueName: \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.544095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.543982    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.646710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.646369    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.646710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.646427    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdj2\" (UniqueName: \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.646710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.646518    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.649408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.649382    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.675552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.671293    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdj2\" (UniqueName: \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.681668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.679628    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e3e294-frqd7\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.743886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.743857    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:23:48.788627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.788596    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:23:48.799701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.799667    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:48.856594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.854101    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\") pod \"11f91250-9849-4f75-82b2-833a1b476bf0\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") "
Nov 03 20:23:48.856594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.854454    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\") pod \"acb83283-8519-47ab-b5c9-c213b417d726\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") "
Nov 03 20:23:48.856594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.854491    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\") pod \"11f91250-9849-4f75-82b2-833a1b476bf0\" (UID: \"11f91250-9849-4f75-82b2-833a1b476bf0\") "
Nov 03 20:23:48.856594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.854590    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\" (UniqueName: \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\") pod \"acb83283-8519-47ab-b5c9-c213b417d726\" (UID: \"acb83283-8519-47ab-b5c9-c213b417d726\") "
Nov 03 20:23:48.869560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.864796    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume" pod "acb83283-8519-47ab-b5c9-c213b417d726" (UID: "acb83283-8519-47ab-b5c9-c213b417d726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:48.869560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.865263    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume" pod "11f91250-9849-4f75-82b2-833a1b476bf0" (UID: "11f91250-9849-4f75-82b2-833a1b476bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:48.876754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.876710    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86" pod "acb83283-8519-47ab-b5c9-c213b417d726" (UID: "acb83283-8519-47ab-b5c9-c213b417d726"). InnerVolumeSpecName "kube-api-access-9sk86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:48.885707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.881657    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl" pod "11f91250-9849-4f75-82b2-833a1b476bf0" (UID: "11f91250-9849-4f75-82b2-833a1b476bf0"). InnerVolumeSpecName "kube-api-access-2w5cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:48.957425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.957359    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb83283-8519-47ab-b5c9-c213b417d726-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:48.957425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.957385    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5cl\" (UniqueName: \"kubernetes.io/projected/11f91250-9849-4f75-82b2-833a1b476bf0-kube-api-access-2w5cl\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:48.957425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.957395    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sk86\" (UniqueName: \"kubernetes.io/projected/acb83283-8519-47ab-b5c9-c213b417d726-kube-api-access-9sk86\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:48.957425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.957404    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f91250-9849-4f75-82b2-833a1b476bf0-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:48.971950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.971881    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:23:48.973135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973083    5424 cpu_manager.go:414] "containerMap: removing container" podUID="acb83283-8519-47ab-b5c9-c213b417d726" containerName="payload-processing"
Nov 03 20:23:48.973135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973114    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb83283-8519-47ab-b5c9-c213b417d726" containerName="payload-processing"
Nov 03 20:23:48.973345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973166    5424 cpu_manager.go:414] "containerMap: removing container" podUID="11f91250-9849-4f75-82b2-833a1b476bf0" containerName="payload-pre-processing"
Nov 03 20:23:48.973345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973175    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f91250-9849-4f75-82b2-833a1b476bf0" containerName="payload-pre-processing"
Nov 03 20:23:48.973449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973378    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f91250-9849-4f75-82b2-833a1b476bf0" containerName="payload-pre-processing"
Nov 03 20:23:48.973449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.973400    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb83283-8519-47ab-b5c9-c213b417d726" containerName="payload-processing"
Nov 03 20:23:49.000551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.989083    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.000551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:48.997002    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:23:49.011373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.010306    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-503c4e-openshift-default-dockercfg-qclwv\""
Nov 03 20:23:49.028592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.027642    5424 generic.go:356] "Generic (PLEG): container finished" podID="acb83283-8519-47ab-b5c9-c213b417d726" containerID="b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181" exitCode=0
Nov 03 20:23:49.028592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.027931    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" event={"ID":"acb83283-8519-47ab-b5c9-c213b417d726","Type":"ContainerDied","Data":"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"}
Nov 03 20:23:49.028592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.027991    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg" event={"ID":"acb83283-8519-47ab-b5c9-c213b417d726","Type":"ContainerDied","Data":"1c65a4829a85219c5ec8ead907cb411fe2b01ad7424fd1e137917bb73b6aaab1"}
Nov 03 20:23:49.028592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.028024    5424 scope.go:122] "RemoveContainer" containerID="b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"
Nov 03 20:23:49.028592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.028190    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"
Nov 03 20:23:49.037543 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.037494    5424 generic.go:356] "Generic (PLEG): container finished" podID="11f91250-9849-4f75-82b2-833a1b476bf0" containerID="f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6" exitCode=0
Nov 03 20:23:49.037692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.037596    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" event={"ID":"11f91250-9849-4f75-82b2-833a1b476bf0","Type":"ContainerDied","Data":"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"}
Nov 03 20:23:49.037692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.037646    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f" event={"ID":"11f91250-9849-4f75-82b2-833a1b476bf0","Type":"ContainerDied","Data":"76fdb4d0f680285feda03528cb3c7b0c1eece868e9e9fc2e18561e4d8296aa5e"}
Nov 03 20:23:49.037838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.037730    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"
Nov 03 20:23:49.049029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.049004    5424 scope.go:122] "RemoveContainer" containerID="b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"
Nov 03 20:23:49.049405 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:49.049377    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181\": container with ID starting with b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181 not found: ID does not exist" containerID="b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"
Nov 03 20:23:49.049491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.049420    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181"} err="failed to get container status \"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181\": rpc error: code = NotFound desc = could not find container \"b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181\": container with ID starting with b2a5ba11fa2a7e8f619f46e242f53d487650866fff06dc2e601d22467fc98181 not found: ID does not exist"
Nov 03 20:23:49.049491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.049446    5424 scope.go:122] "RemoveContainer" containerID="f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"
Nov 03 20:23:49.064137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.063565    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.064137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.063801    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.064137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.063896    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzlb\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.064137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.064035    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.066643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.066179    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.066643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.066284    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.066643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.066404    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.066643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.066564    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.066643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.066603    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.071919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.071903    5424 scope.go:122] "RemoveContainer" containerID="f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"
Nov 03 20:23:49.073441 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:23:49.072343    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6\": container with ID starting with f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6 not found: ID does not exist" containerID="f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"
Nov 03 20:23:49.073621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.073589    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6"} err="failed to get container status \"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6\": rpc error: code = NotFound desc = could not find container \"f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6\": container with ID starting with f4e8862f827043cea026c017aae527a302deb1a072fbe518b0dfbcf06ac933e6 not found: ID does not exist"
Nov 03 20:23:49.114636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.114600    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:23:49.126392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.126361    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a619f336-5ff8fd94b-d6nrg"]
Nov 03 20:23:49.162052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.162022    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:23:49.172934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.172786    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"]
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176011    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176057    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176100    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176188    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176250    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzlb\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176291    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176320    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176386    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.176497    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.177137    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.177356    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.178816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.178381    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.179325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.179182    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.180809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.180784    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.182366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.182346    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.193843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.179437    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.198200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.198173    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.198298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.198238    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzlb\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\") pod \"e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") " pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.199023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.199002    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a619f336-54964bc959-9h87f"]
Nov 03 20:23:49.319585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.315782    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:49.866221 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:49.866192    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe409dc7_7754_40d7_994a_5e92275bef6f.slice/crio-e4b2b7348dc8708b8c705cb5c17972051b9f1368fb598f998e1a1bdd5d0e7e70 WatchSource:0}: Error finding container e4b2b7348dc8708b8c705cb5c17972051b9f1368fb598f998e1a1bdd5d0e7e70: Status 404 returned error can't find the container with id e4b2b7348dc8708b8c705cb5c17972051b9f1368fb598f998e1a1bdd5d0e7e70
Nov 03 20:23:49.868214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.868188    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:23:49.869851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.869811    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:49.869934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.869898    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:49.869980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.869939    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:49.922329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.922295    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f91250-9849-4f75-82b2-833a1b476bf0" path="/var/lib/kubelet/pods/11f91250-9849-4f75-82b2-833a1b476bf0/volumes"
Nov 03 20:23:49.922906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:49.922884    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb83283-8519-47ab-b5c9-c213b417d726" path="/var/lib/kubelet/pods/acb83283-8519-47ab-b5c9-c213b417d726/volumes"
Nov 03 20:23:50.053180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.053139    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerStarted","Data":"39a92da73e366ef720ee10eeaf70de9fa0ef59c5a73907d98b09425483d4aa2d"}
Nov 03 20:23:50.053382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.053195    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerStarted","Data":"e4b2b7348dc8708b8c705cb5c17972051b9f1368fb598f998e1a1bdd5d0e7e70"}
Nov 03 20:23:50.058009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.057973    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7" event={"ID":"ffff2e08-dc84-43a7-9c17-6ac22200a3e8","Type":"ContainerStarted","Data":"ddef6513cd4d4df8428efdcaf87c805c592cc51a1f68294eac1a7af1230dfb6a"}
Nov 03 20:23:50.058186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.058031    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7" event={"ID":"ffff2e08-dc84-43a7-9c17-6ac22200a3e8","Type":"ContainerStarted","Data":"df662284b85f4c1dd3edf069301ec654b6ed322a5de4a939a7cfd58637fcb539"}
Nov 03 20:23:50.097645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.097594    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podStartSLOduration=2.097578083 podStartE2EDuration="2.097578083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:50.096568986 +0000 UTC m=+1172.305584977" watchObservedRunningTime="2026-11-03 20:23:50.097578083 +0000 UTC m=+1172.306594079"
Nov 03 20:23:50.139157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.139046    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7" podStartSLOduration=2.139033098 podStartE2EDuration="2.139033098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:50.133349817 +0000 UTC m=+1172.342365812" watchObservedRunningTime="2026-11-03 20:23:50.139033098 +0000 UTC m=+1172.348049093"
Nov 03 20:23:50.316147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.316103    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:50.318311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.318283    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:23:50.318455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:50.318333    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:23:51.097250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.097216    5424 generic.go:356] "Generic (PLEG): container finished" podID="ffff2e08-dc84-43a7-9c17-6ac22200a3e8" containerID="ddef6513cd4d4df8428efdcaf87c805c592cc51a1f68294eac1a7af1230dfb6a" exitCode=0
Nov 03 20:23:51.097729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.097297    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7" event={"ID":"ffff2e08-dc84-43a7-9c17-6ac22200a3e8","Type":"ContainerDied","Data":"ddef6513cd4d4df8428efdcaf87c805c592cc51a1f68294eac1a7af1230dfb6a"}
Nov 03 20:23:51.187832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.187796    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:23:51.193114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.193084    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.213587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.209735    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-17b3d9-openshift-default-dockercfg-z7nlh\""
Nov 03 20:23:51.229578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.229497    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:23:51.293291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293258    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293322    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgvq\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293437    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293487    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293515    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293577    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293735    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293789    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.293948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.293931    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.396623    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.396674    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.396768    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.396960    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397022    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgvq\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397067    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397100    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397127    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397167    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.397975    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.398583    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.399115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.398820    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.422431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.421701    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.422431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.422291    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.442552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.427704    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.442552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.439149    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.451124 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.451092    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgvq\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.461316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.453151    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\") pod \"e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") " pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.505896 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.505864    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:51.521953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.521925    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:51.979856 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:23:51.979823    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a1ff7d_4d50_4776_ac88_70ab88414d78.slice/crio-ddbe45d95d275b483260f6935c27d6610c9dde3f3d1cc2be60c6ca75ca140777 WatchSource:0}: Error finding container ddbe45d95d275b483260f6935c27d6610c9dde3f3d1cc2be60c6ca75ca140777: Status 404 returned error can't find the container with id ddbe45d95d275b483260f6935c27d6610c9dde3f3d1cc2be60c6ca75ca140777
Nov 03 20:23:51.980508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.980485    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:23:51.983507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.983473    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:51.983690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.983608    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:51.983690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:51.983646    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:52.110349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.110311    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerStarted","Data":"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"}
Nov 03 20:23:52.110906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.110380    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerStarted","Data":"ddbe45d95d275b483260f6935c27d6610c9dde3f3d1cc2be60c6ca75ca140777"}
Nov 03 20:23:52.113791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.111131    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:52.114822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.114659    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:23:52.225148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.221839    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podStartSLOduration=1.221816613 podStartE2EDuration="1.221816613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:52.169551554 +0000 UTC m=+1174.378567574" watchObservedRunningTime="2026-11-03 20:23:52.221816613 +0000 UTC m=+1174.430832610"
Nov 03 20:23:52.396353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.396322    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:52.468898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.462418    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\") pod \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") "
Nov 03 20:23:52.468898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.462594    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\" (UniqueName: \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\") pod \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") "
Nov 03 20:23:52.468898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.465880    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca" pod "ffff2e08-dc84-43a7-9c17-6ac22200a3e8" (UID: "ffff2e08-dc84-43a7-9c17-6ac22200a3e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:52.468898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.466378    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\") pod \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\" (UID: \"ffff2e08-dc84-43a7-9c17-6ac22200a3e8\") "
Nov 03 20:23:52.468898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.467697    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:52.479271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.479232    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca" pod "ffff2e08-dc84-43a7-9c17-6ac22200a3e8" (UID: "ffff2e08-dc84-43a7-9c17-6ac22200a3e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:52.479441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.479400    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2" pod "ffff2e08-dc84-43a7-9c17-6ac22200a3e8" (UID: "ffff2e08-dc84-43a7-9c17-6ac22200a3e8"). InnerVolumeSpecName "kube-api-access-bqdj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:52.522550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.522496    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:52.569046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.569011    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:52.569046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:52.569050    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdj2\" (UniqueName: \"kubernetes.io/projected/ffff2e08-dc84-43a7-9c17-6ac22200a3e8-kube-api-access-bqdj2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:23:53.121816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:53.121784    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"
Nov 03 20:23:53.121816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:53.121801    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7" event={"ID":"ffff2e08-dc84-43a7-9c17-6ac22200a3e8","Type":"ContainerDied","Data":"df662284b85f4c1dd3edf069301ec654b6ed322a5de4a939a7cfd58637fcb539"}
Nov 03 20:23:53.122307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:53.121857    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df662284b85f4c1dd3edf069301ec654b6ed322a5de4a939a7cfd58637fcb539"
Nov 03 20:23:53.530023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:53.529990    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:54.130907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:54.130865    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:23:54.131832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:23:54.131808    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:24:12.289017 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:24:12.288937    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:24:18.379167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.379122    5424 generic.go:356] "Generic (PLEG): container finished" podID="018e100d-188f-4968-ad37-869b18b33f59" containerID="8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10" exitCode=0
Nov 03 20:24:18.379781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.379277    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerDied","Data":"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"}
Nov 03 20:24:18.380132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.380113    5424 scope.go:122] "RemoveContainer" containerID="8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"
Nov 03 20:24:18.382668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.382321    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:18.382668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.382414    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:18.382668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:18.382447    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:19.347237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.347202    5424 scope.go:122] "RemoveContainer" containerID="2fe7da4f7c829c9e4c4df2191e78918b287d0bd25728c906d09a8e2c83d560ac"
Nov 03 20:24:19.358993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.358970    5424 scope.go:122] "RemoveContainer" containerID="59c01b0bbdff32c6953c578348348966adada897620630161d9e90dba0cdb680"
Nov 03 20:24:19.370301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.370274    5424 scope.go:122] "RemoveContainer" containerID="32f4478f7e3085ddba07ad267fb1431ef807a6edfe3153ae3052000697174d8e"
Nov 03 20:24:19.387183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.387155    5424 scope.go:122] "RemoveContainer" containerID="469dfa46bad7966ebe23064b74bde09e6ff0357d9930e2fcb3d5204a1c0062c4"
Nov 03 20:24:19.396840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.396800    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerStarted","Data":"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"}
Nov 03 20:24:19.399893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.399872    5424 scope.go:122] "RemoveContainer" containerID="115425ba0f028ebf0dbc03474e25fdf1eef555d1e1aae460969cf045d596e16d"
Nov 03 20:24:19.410720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.410699    5424 scope.go:122] "RemoveContainer" containerID="16856a3c2182e1d9fddd43fc7b46345cb166d559c5d16037d2493643c2a884c7"
Nov 03 20:24:19.445403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.445372    5424 scope.go:122] "RemoveContainer" containerID="faef4cefcf22762fdf6e1b84b40c6fd09a23773b61b762bb8e84f2872495288c"
Nov 03 20:24:19.457370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.457349    5424 scope.go:122] "RemoveContainer" containerID="53d00490ad5b25755a300726af5b1ee2251d7bf6871a3dd0bf2ad6383c2f84f8"
Nov 03 20:24:19.468333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.468308    5424 scope.go:122] "RemoveContainer" containerID="306fcfe19f89a3806ad221a5d8cc2c2ce56fd3c0a6a0b989e20ba367b8290480"
Nov 03 20:24:19.479719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.479694    5424 scope.go:122] "RemoveContainer" containerID="09a30153287205f33e81632bde9e70fa6e3b4dca96ec911338c82bbc7d4aa586"
Nov 03 20:24:19.491058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.491033    5424 scope.go:122] "RemoveContainer" containerID="17f1a703a877c7d5b2d7f2b5d66bd9a838f56332e2fc832055bef87cc3eb5495"
Nov 03 20:24:19.501804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.501781    5424 scope.go:122] "RemoveContainer" containerID="6f0528363324c96c7c8567bcd52ed61e39bbbacd6901b1d2cac08ee1c63a5610"
Nov 03 20:24:19.798634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.798591    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:24:19.800109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.800076    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 20:24:19.800233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:19.800146    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 20:24:20.798742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:20.798706    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 20:24:20.799154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:20.798776    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 20:24:21.797967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:21.797932    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 20:24:21.798288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:21.798051    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 20:24:22.798866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:22.798827    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 20:24:22.799294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:22.798885    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 20:24:23.798155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:23.798116    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 20:24:23.798359 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:23.798182    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 20:24:24.408234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:24.408204    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:24:24.802787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:24.802758    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:24:24.803974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:24.803950    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:24:32.868766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.868727    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:24:32.869436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.869419    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ffff2e08-dc84-43a7-9c17-6ac22200a3e8" containerName="revoke-keys"
Nov 03 20:24:32.869499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.869441    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffff2e08-dc84-43a7-9c17-6ac22200a3e8" containerName="revoke-keys"
Nov 03 20:24:32.869602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.869591    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffff2e08-dc84-43a7-9c17-6ac22200a3e8" containerName="revoke-keys"
Nov 03 20:24:32.876774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.876756    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:32.890255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.890230    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:24:32.896817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.896787    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-e3ba87\""
Nov 03 20:24:32.904120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.904097    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-e3ba87-dockercfg-d52q6\""
Nov 03 20:24:32.955001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.954972    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88kt\" (UniqueName: \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:32.955235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:32.955198    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.064554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.062715    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.064554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.062927    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88kt\" (UniqueName: \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.109561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.106744    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.150502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.150413    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88kt\" (UniqueName: \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\") pod \"payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.204638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.192832    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:33.559596 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:24:33.559556    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9077b4af_e665_4e50_b5d6_eff8cdb2e839.slice/crio-8bd2432fd0f1785fdd83f0f62b5867baab5e893b7b964351c875bbb4b3c2e275 WatchSource:0}: Error finding container 8bd2432fd0f1785fdd83f0f62b5867baab5e893b7b964351c875bbb4b3c2e275: Status 404 returned error can't find the container with id 8bd2432fd0f1785fdd83f0f62b5867baab5e893b7b964351c875bbb4b3c2e275
Nov 03 20:24:33.563495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.562714    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:24:33.621605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:33.621570    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" event={"ID":"9077b4af-e665-4e50-b5d6-eff8cdb2e839","Type":"ContainerStarted","Data":"8bd2432fd0f1785fdd83f0f62b5867baab5e893b7b964351c875bbb4b3c2e275"}
Nov 03 20:24:34.635317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:34.635273    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" event={"ID":"9077b4af-e665-4e50-b5d6-eff8cdb2e839","Type":"ContainerStarted","Data":"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"}
Nov 03 20:24:34.635899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:34.635373    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:34.676626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:34.676559    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" podStartSLOduration=2.676518835 podStartE2EDuration="2.676518835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:34.672872504 +0000 UTC m=+1216.881888500" watchObservedRunningTime="2026-11-03 20:24:34.676518835 +0000 UTC m=+1216.885534835"
Nov 03 20:24:39.834425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.834389    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:24:39.837773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.837752    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:39.855203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.855174    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:24:39.861766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.861738    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a0866c\""
Nov 03 20:24:39.907382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.907348    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqw8l\" (UniqueName: \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:39.907575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.907464    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:39.920763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.920738    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:24:39.924038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.924022    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:39.944309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.941398    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:24:39.953593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:39.944937    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a0866c-dockercfg-6vg8w\""
Nov 03 20:24:40.021552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.010669    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqw8l\" (UniqueName: \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.021552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.010730    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bswk\" (UniqueName: \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.021552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.010771    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.021552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.010930    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.021552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.012281    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.061608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.056045    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqw8l\" (UniqueName: \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\") pod \"payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.115039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.114936    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bswk\" (UniqueName: \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.115039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.115007    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.122332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.116157    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.137758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.136308    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bswk\" (UniqueName: \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\") pod \"payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.151836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.151799    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.243168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.243135    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.483399 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:24:40.483365    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c028d5d_bdea_479d_a0cf_2e97b902dd46.slice/crio-bdd5eb6cf3761ad12759a287a798a041010f2099d487b6fb403bb0a84daa61a0 WatchSource:0}: Error finding container bdd5eb6cf3761ad12759a287a798a041010f2099d487b6fb403bb0a84daa61a0: Status 404 returned error can't find the container with id bdd5eb6cf3761ad12759a287a798a041010f2099d487b6fb403bb0a84daa61a0
Nov 03 20:24:40.486440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.486415    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:24:40.586913 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:24:40.586876    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3038f1e_ef26_48c6_8f16_22a1fbcbfb43.slice/crio-f15a5fa260b5d57f04f41a85bdac6f292eaa7ac8d46f882411aa6ebdb31a66a2 WatchSource:0}: Error finding container f15a5fa260b5d57f04f41a85bdac6f292eaa7ac8d46f882411aa6ebdb31a66a2: Status 404 returned error can't find the container with id f15a5fa260b5d57f04f41a85bdac6f292eaa7ac8d46f882411aa6ebdb31a66a2
Nov 03 20:24:40.589242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.589218    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:24:40.713573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.713510    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" event={"ID":"7c028d5d-bdea-479d-a0cf-2e97b902dd46","Type":"ContainerStarted","Data":"1d25c64e3c238c13b9a4de64ad6c4d937113dffbd518c32bd9319c1d7425a3fb"}
Nov 03 20:24:40.713779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.713600    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" event={"ID":"7c028d5d-bdea-479d-a0cf-2e97b902dd46","Type":"ContainerStarted","Data":"bdd5eb6cf3761ad12759a287a798a041010f2099d487b6fb403bb0a84daa61a0"}
Nov 03 20:24:40.713779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.713649    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:40.715679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.715650    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" event={"ID":"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43","Type":"ContainerStarted","Data":"ac486fdfc98117f7896bf7e3b9c440be2269a83305bd6c8df8d92b352c69fc41"}
Nov 03 20:24:40.715841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.715697    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" event={"ID":"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43","Type":"ContainerStarted","Data":"f15a5fa260b5d57f04f41a85bdac6f292eaa7ac8d46f882411aa6ebdb31a66a2"}
Nov 03 20:24:40.715841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.715724    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:40.754579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.754480    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" podStartSLOduration=1.7544582100000001 podStartE2EDuration="1.75445821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:40.75313552 +0000 UTC m=+1222.962151528" watchObservedRunningTime="2026-11-03 20:24:40.75445821 +0000 UTC m=+1222.963474208"
Nov 03 20:24:40.791092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:40.791039    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" podStartSLOduration=1.791025478 podStartE2EDuration="1.791025478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:40.790757466 +0000 UTC m=+1222.999773480" watchObservedRunningTime="2026-11-03 20:24:40.791025478 +0000 UTC m=+1223.000041475"
Nov 03 20:24:42.286872 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:24:42.286833    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:24:43.367560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.351669    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:24:43.374917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.374888    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.397626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.397578    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:24:43.405243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.405210    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eb87f429-dockercfg-fxjz4\""
Nov 03 20:24:43.407614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.407491    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.408085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.408057    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6x82\" (UniqueName: \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.450574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.450003    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eb87f429\""
Nov 03 20:24:43.512505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.511971    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6x82\" (UniqueName: \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.512505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.512129    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.513133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.513104    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.543729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.543691    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6x82\" (UniqueName: \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:43.810587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:43.810489    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:44.116310 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:24:44.116277    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7291627_dc61_43d8_a726_4272310af44e.slice/crio-4d9e0744dcf6934a77f2f0132a3d32d31bed24c2325d31c63e9cd136aab8a0bf WatchSource:0}: Error finding container 4d9e0744dcf6934a77f2f0132a3d32d31bed24c2325d31c63e9cd136aab8a0bf: Status 404 returned error can't find the container with id 4d9e0744dcf6934a77f2f0132a3d32d31bed24c2325d31c63e9cd136aab8a0bf
Nov 03 20:24:44.119540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:44.119408    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:24:44.764047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:44.764015    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" event={"ID":"b7291627-dc61-43d8-a726-4272310af44e","Type":"ContainerStarted","Data":"c91a25952376799fc69aac1125a1578d5dd6dffe208044583822cb68e6cd496b"}
Nov 03 20:24:44.764503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:44.764067    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" event={"ID":"b7291627-dc61-43d8-a726-4272310af44e","Type":"ContainerStarted","Data":"4d9e0744dcf6934a77f2f0132a3d32d31bed24c2325d31c63e9cd136aab8a0bf"}
Nov 03 20:24:44.764503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:44.764116    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:44.805556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:44.805477    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" podStartSLOduration=1.805461766 podStartE2EDuration="1.805461766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:44.801950226 +0000 UTC m=+1227.010966222" watchObservedRunningTime="2026-11-03 20:24:44.805461766 +0000 UTC m=+1227.014477783"
Nov 03 20:24:45.645899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:45.645874    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:24:51.027619 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:51.027580    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"]
Nov 03 20:24:51.039829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:51.039797    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb8b2133-vws4b"]
Nov 03 20:24:51.727692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:51.727663    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:24:51.727908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:51.727743    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:24:51.923952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:51.923920    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acb4e95-17c7-4acd-8825-c6654363b37c" path="/var/lib/kubelet/pods/4acb4e95-17c7-4acd-8825-c6654363b37c/volumes"
Nov 03 20:24:54.880463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.880433    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"]
Nov 03 20:24:54.886317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.886288    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:54.903977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.903944    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"]
Nov 03 20:24:54.909636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.909584    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:24:54.919284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.909701    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:24:54.919284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.909767    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:24:54.919284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:54.912790    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:24:55.042748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.042710    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.042961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.042923    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7mz\" (UniqueName: \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.043205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.043174    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.144414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.144309    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.144650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.144434    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.144650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.144624    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7mz\" (UniqueName: \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.145112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.145089    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.146965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.146945    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.164023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.163991    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7mz\" (UniqueName: \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\") pod \"maas-api-revoke-sub-1eae31f9-dd4p9\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.206119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.206083    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:24:55.531710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.531682    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"]
Nov 03 20:24:55.532136 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:24:55.532113    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c31676_ec28_4946_97da_6551112ca316.slice/crio-01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd WatchSource:0}: Error finding container 01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd: Status 404 returned error can't find the container with id 01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd
Nov 03 20:24:55.782308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.781852    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:24:55.884862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:55.884817    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9" event={"ID":"09c31676-ec28-4946-97da-6551112ca316","Type":"ContainerStarted","Data":"01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd"}
Nov 03 20:24:56.931814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:56.931515    5424 generic.go:356] "Generic (PLEG): container finished" podID="09c31676-ec28-4946-97da-6551112ca316" containerID="5638c17d84dc28791192fd9a249d6d3e3cd9398a67a8fb85b389d4071de4f377" exitCode=0
Nov 03 20:24:56.931814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:56.931662    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9" event={"ID":"09c31676-ec28-4946-97da-6551112ca316","Type":"ContainerDied","Data":"5638c17d84dc28791192fd9a249d6d3e3cd9398a67a8fb85b389d4071de4f377"}
Nov 03 20:24:57.949555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:57.945818    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:57.983564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:57.978621    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.263807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.263719    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.427690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.427628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.428314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.428282    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.455696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.455629    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.505655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.486638    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-ctrtq" podUID="2cb0b7a8-707d-4413-b244-3e2f313b0e48" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.506498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.506447    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.528587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.528516    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Nov 03 20:24:58.529141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.529108    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.591566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.587625    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.591566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.588746    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.615618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.614625    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.662558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.656658    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.681567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.679630    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.703570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.698625    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.742704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.731638    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:58.862616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.857695    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 20:24:58.862616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:58.858056    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.369687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.369638    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.600039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.402202    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:24:59.617234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.430921    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.618577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.484711    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.619796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.485774    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:24:59.626457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.595847    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.626457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.595892    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:24:59.626457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.597211    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.599722    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.600302    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630419    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630477    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630519    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630600    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630658    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630710    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630746    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:24:59.642015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.630779    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:24:59.670260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.670221    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.672106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.670295    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.688762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.688730    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:24:59.702584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.691333    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.727096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.725364    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.763622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.730462    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:24:59.763622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.744025    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:24:59.763622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.752905    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:24:59.782576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.780372    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:24:59.818553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.818267    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:24:59.903610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:24:59.858777    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.190575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.190456    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.303558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.299471    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:25:00.360944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.360909    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:25:00.395718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.372182    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:25:00.576709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.571502    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\" (UniqueName: \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\") pod \"09c31676-ec28-4946-97da-6551112ca316\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") "
Nov 03 20:25:00.576709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.571592    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\") pod \"09c31676-ec28-4946-97da-6551112ca316\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") "
Nov 03 20:25:00.576709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.571723    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\") pod \"09c31676-ec28-4946-97da-6551112ca316\" (UID: \"09c31676-ec28-4946-97da-6551112ca316\") "
Nov 03 20:25:00.604579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.604063    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.610697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.610662    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca" pod "09c31676-ec28-4946-97da-6551112ca316" (UID: "09c31676-ec28-4946-97da-6551112ca316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:00.624893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.624366    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.645562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.624417    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.646174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.624439    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.699988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.624459    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.700847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.624476    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.704555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.651595    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.803862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.803831    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:25:00.832407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.659992    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c31676-ec28-4946-97da-6551112ca316-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:00.832668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.678434    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.833225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.678572    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.835325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.730770    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.835844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.754976    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.836577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.651628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.889609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.884893    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.952396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:00.952367    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:25:01.013095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.013058    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz" pod "09c31676-ec28-4946-97da-6551112ca316" (UID: "09c31676-ec28-4946-97da-6551112ca316"). InnerVolumeSpecName "kube-api-access-6j7mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:01.021813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.021751    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca" pod "09c31676-ec28-4946-97da-6551112ca316" (UID: "09c31676-ec28-4946-97da-6551112ca316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:01.060607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.057706    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.083067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.082970    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7mz\" (UniqueName: \"kubernetes.io/projected/09c31676-ec28-4946-97da-6551112ca316-kube-api-access-6j7mz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:01.083332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.083309    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c31676-ec28-4946-97da-6551112ca316-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:01.234744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.234691    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.571771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.571728    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9" event={"ID":"09c31676-ec28-4946-97da-6551112ca316","Type":"ContainerDied","Data":"01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd"}
Nov 03 20:25:01.602770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.602727    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e3fd814d0c26c02112b029861bcce4f2b87ed0ed3750c65237f93cbe6bf7bd"
Nov 03 20:25:01.602996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.582886    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"
Nov 03 20:25:01.644555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.618608    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.646225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.646186    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.739470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.704629    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.739470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.705006    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.739470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.706300    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.831822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.830257    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.837305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.837265    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.838430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.838396    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.841004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.838877    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:25:01.894580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.886935    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.922120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.922081    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:01.938681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.938650    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:25:01.954603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:01.953859    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.153610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.100476    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:25:02.307011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.155262    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 20:25:02.328474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.324625    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.620256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.619236    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.660552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.659251    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.678575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.673710    5424 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Nov 03 20:25:02.714815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.707288    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.714815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.707733    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:25:02.760824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.759067    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:25:02.853580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.834181    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:25:02.853580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.850637    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.853580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.851070    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:02.853580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.851354    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:25:02.853580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.851641    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:25:02.959397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.959303    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:02.965273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:02.965248    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:25:03.310257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.310214    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:03.362558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.361362    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:03.502303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.502274    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:25:03.505251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.505219    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:25:03.517553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.509793    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:25:03.633610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.632882    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 20:25:03.670592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.659585    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:03.748695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.748663    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:25:03.838556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.755826    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:25:03.838556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:03.764451    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 20:25:04.081629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.069701    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:25:04.208627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.190325    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:25:04.208627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.193110    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:25:04.268256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.268216    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:25:04.396643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.364596    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:04.623985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.623955    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:25:04.662591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.662475    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:04.798566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.797647    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:04.802106 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.802058    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:04.881318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:04.881273    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:05.116756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:05.116720    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"]
Nov 03 20:25:05.129617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:05.129580    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82271bb2-dqnss"]
Nov 03 20:25:05.281564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:05.270639    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:05.610609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:05.610521    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:25:05.949217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:05.949134    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69aa372d-e9ff-4dfe-90de-7c24e9f4908e" path="/var/lib/kubelet/pods/69aa372d-e9ff-4dfe-90de-7c24e9f4908e/volumes"
Nov 03 20:25:08.152821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:08.152781    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:08.153329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:08.152839    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:10.172717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:10.172673    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:10.173134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:10.172745    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:11.033978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:11.033940    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"]
Nov 03 20:25:11.521892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:11.046013    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0f0c22-w25v6"]
Nov 03 20:25:11.934618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:11.934459    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceafee3-ff21-434e-8bd0-85e17341e648" path="/var/lib/kubelet/pods/7ceafee3-ff21-434e-8bd0-85e17341e648/volumes"
Nov 03 20:25:12.198466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.198381    5424 generic.go:356] "Generic (PLEG): container finished" podID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerID="39a92da73e366ef720ee10eeaf70de9fa0ef59c5a73907d98b09425483d4aa2d" exitCode=0
Nov 03 20:25:12.198466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.198457    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerDied","Data":"39a92da73e366ef720ee10eeaf70de9fa0ef59c5a73907d98b09425483d4aa2d"}
Nov 03 20:25:12.198976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.198957    5424 scope.go:122] "RemoveContainer" containerID="39a92da73e366ef720ee10eeaf70de9fa0ef59c5a73907d98b09425483d4aa2d"
Nov 03 20:25:12.200411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200386    5424 generic.go:356] "Generic (PLEG): container finished" podID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerID="6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d" exitCode=0
Nov 03 20:25:12.200523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200454    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerDied","Data":"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"}
Nov 03 20:25:12.200847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200816    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.200895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200876    5424 scope.go:122] "RemoveContainer" containerID="6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"
Nov 03 20:25:12.200937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200911    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.200975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.200959    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.202433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202414    5424 generic.go:356] "Generic (PLEG): container finished" podID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerID="aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a" exitCode=0
Nov 03 20:25:12.202517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202487    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerDied","Data":"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"}
Nov 03 20:25:12.202608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202584    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.202664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202650    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.202699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202685    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.202942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.202926    5424 scope.go:122] "RemoveContainer" containerID="aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"
Nov 03 20:25:12.204961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.204862    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.204961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.204925    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.205059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.204969    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.310836 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:12.309790    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:25:12.962781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:12.962734    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:25:13.213435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.213355    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerStarted","Data":"3819ff3a32a94469de70be4b437abc762d1b197b83b2921ca5d8d9205f09c325"}
Nov 03 20:25:13.216588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.216549    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerStarted","Data":"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"}
Nov 03 20:25:13.244569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.239979    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerStarted","Data":"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"}
Nov 03 20:25:13.315933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.315896    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:25:13.316140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.316122    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:13.316191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.316166    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:13.530555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.523956    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:25:13.530555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.524086    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:13.530555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:13.524131    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:14.046553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.045992    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:25:14.046553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.046129    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:25:14.046553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.046180    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:25:14.343748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.343653    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:14.344053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.344024    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:14.543564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.537982    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:14.543564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:14.538063    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:15.047565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.047109    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:25:15.047565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.047173    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:25:15.316930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.316844    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:15.317076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.316930    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:15.559557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.531996    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:15.559557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:15.532060    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:16.070558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.049561    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:25:16.070558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.049628    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:25:16.316162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.316126    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:16.316377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.316189    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:16.539558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.525984    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:16.539558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:16.526049    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:17.043553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.042991    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:25:17.043553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.043068    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:25:17.317632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.316943    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:17.317632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.317011    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:17.542319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.542275    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:17.542669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:17.542634    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:18.044751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.044714    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:25:18.045042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.045015    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:25:18.336063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.335969    5424 patch_prober.go:29] interesting pod/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:25:18.336655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.336620    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:25:18.529557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.528920    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:25:18.529557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:18.528999    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 20:25:19.063213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.063180    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"]
Nov 03 20:25:19.072870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.072833    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9844baaf-lswrb"]
Nov 03 20:25:19.241072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.238736    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:25:19.241072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.240402    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:25:19.241847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.241826    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:25:19.357372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.357280    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:25:19.360741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.359956    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:25:19.389987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.380822    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:25:19.552960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.552924    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:25:19.556370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.556348    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:25:19.557598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.557579    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:25:19.698375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.698287    5424 scope.go:122] "RemoveContainer" containerID="41aa8b8f40ee4e54e75540f00776a2a542195d55f6548464fce5d42cb781e25b"
Nov 03 20:25:19.923393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:19.923355    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9" path="/var/lib/kubelet/pods/c5e13f9b-b53c-4f2e-9d4d-5fe02dac08c9/volumes"
Nov 03 20:25:23.041405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:23.041373    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"]
Nov 03 20:25:23.049224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:23.049185    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f61ec-q9dz9"]
Nov 03 20:25:23.921489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:23.921456    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fcd831-ce34-4eaa-a781-e226ec917203" path="/var/lib/kubelet/pods/01fcd831-ce34-4eaa-a781-e226ec917203/volumes"
Nov 03 20:25:26.029686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:26.029641    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"]
Nov 03 20:25:26.041230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:26.041201    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079b4967-mn55l"]
Nov 03 20:25:27.921763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:27.921677    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95216b61-05d8-4878-a08d-16ce63a3520a" path="/var/lib/kubelet/pods/95216b61-05d8-4878-a08d-16ce63a3520a/volumes"
Nov 03 20:25:29.640898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.640865    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-2c8921"]
Nov 03 20:25:29.642847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.642815    5424 cpu_manager.go:414] "containerMap: removing container" podUID="09c31676-ec28-4946-97da-6551112ca316" containerName="revoke-keys"
Nov 03 20:25:29.643035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.643019    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c31676-ec28-4946-97da-6551112ca316" containerName="revoke-keys"
Nov 03 20:25:29.643346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.643327    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c31676-ec28-4946-97da-6551112ca316" containerName="revoke-keys"
Nov 03 20:25:29.665800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.665768    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-2c8921"]
Nov 03 20:25:29.666164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.666098    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:29.694409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.694379    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2hbv\""
Nov 03 20:25:29.828217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.821021    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:25:29.864305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.864264    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxqx\" (UniqueName: \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\") pod \"test-curl-7024-2c8921\" (UID: \"e7d2ad80-ebd3-462c-b473-ad94d5696fce\") " pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:29.875562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.875511    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:25:29.875914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.875891    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.899422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.899390    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-55b09bd8-openshift-default-dockercfg-2kx4w\""
Nov 03 20:25:29.966634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.966596    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.966861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.966685    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxqx\" (UniqueName: \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\") pod \"test-curl-7024-2c8921\" (UID: \"e7d2ad80-ebd3-462c-b473-ad94d5696fce\") " pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:29.966861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.966783    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.966861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.966835    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967041 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.966929    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.967094    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.967244    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.967312    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.967347    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlsz6\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.967494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.967391    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:29.986297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:29.986263    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxqx\" (UniqueName: \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\") pod \"test-curl-7024-2c8921\" (UID: \"e7d2ad80-ebd3-462c-b473-ad94d5696fce\") " pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:30.068425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068389    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.068649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068469    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.068649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068510    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsz6\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.068810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068787    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.068873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068838    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.068928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068898    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.069004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.068986    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.069068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.069031    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.069122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.069083    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.069286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.069268    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.070311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.070277    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.070805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.070781    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.070981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.070965    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.073556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.073512    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.092238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.092197    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlsz6\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.097064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.097026    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.119983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.119938    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:30.150831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.150732    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.152642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.152616    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\") pod \"e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") " pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.210647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.208580    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:30.481836 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:30.481804    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7d2ad80_ebd3_462c_b473_ad94d5696fce.slice/crio-baf36e5ae4acf32fe8945ca8a0a2fcf6a608377cf118067c862b2571c5b58ef8 WatchSource:0}: Error finding container baf36e5ae4acf32fe8945ca8a0a2fcf6a608377cf118067c862b2571c5b58ef8: Status 404 returned error can't find the container with id baf36e5ae4acf32fe8945ca8a0a2fcf6a608377cf118067c862b2571c5b58ef8
Nov 03 20:25:30.483415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.483383    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-2c8921"]
Nov 03 20:25:30.569755 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:30.569713    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5693a09_6d1f_4d3b_82fc_0354d1193214.slice/crio-02ba6cb8d3991be2dda6a4462b5461a6eba5ffce72fb38155ee7055ad14f143e WatchSource:0}: Error finding container 02ba6cb8d3991be2dda6a4462b5461a6eba5ffce72fb38155ee7055ad14f143e: Status 404 returned error can't find the container with id 02ba6cb8d3991be2dda6a4462b5461a6eba5ffce72fb38155ee7055ad14f143e
Nov 03 20:25:30.571268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.571245    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:25:30.572734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.572701    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:30.572797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.572770    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:30.572840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.572800    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:30.589310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.589279    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerStarted","Data":"02ba6cb8d3991be2dda6a4462b5461a6eba5ffce72fb38155ee7055ad14f143e"}
Nov 03 20:25:30.591172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.591145    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-2c8921" event={"ID":"e7d2ad80-ebd3-462c-b473-ad94d5696fce","Type":"ContainerStarted","Data":"aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619"}
Nov 03 20:25:30.591276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.591186    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-2c8921" event={"ID":"e7d2ad80-ebd3-462c-b473-ad94d5696fce","Type":"ContainerStarted","Data":"baf36e5ae4acf32fe8945ca8a0a2fcf6a608377cf118067c862b2571c5b58ef8"}
Nov 03 20:25:30.629216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:30.629158    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-2c8921" podStartSLOduration=1.629137787 podStartE2EDuration="1.629137787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:30.626146477 +0000 UTC m=+1272.835162475" watchObservedRunningTime="2026-11-03 20:25:30.629137787 +0000 UTC m=+1272.838153785"
Nov 03 20:25:31.512948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.512869    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-2c8921"]
Nov 03 20:25:31.520580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.520546    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-2c8921"]
Nov 03 20:25:31.602346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.602308    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerStarted","Data":"9070150753b7976af16ba5a5ef726f4114def09f91f554f5c9e6ff01124f1606"}
Nov 03 20:25:31.651663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.651591    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podStartSLOduration=2.65157705 podStartE2EDuration="2.65157705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:31.645910433 +0000 UTC m=+1273.854926430" watchObservedRunningTime="2026-11-03 20:25:31.65157705 +0000 UTC m=+1273.860593046"
Nov 03 20:25:31.688405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.688371    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-d75da0"]
Nov 03 20:25:31.689053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.689037    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" containerName="test-curl-7024-2c8921"
Nov 03 20:25:31.689107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.689055    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" containerName="test-curl-7024-2c8921"
Nov 03 20:25:31.689193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.689183    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" containerName="test-curl-7024-2c8921"
Nov 03 20:25:31.691992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.691976    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:31.709583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.709049    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-d75da0"]
Nov 03 20:25:31.715996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.715968    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdq7\" (UniqueName: \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\") pod \"test-curl-7024-d75da0\" (UID: \"9bb252c6-c92b-495e-a55e-4e996fe133f8\") " pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:31.817096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.817010    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdq7\" (UniqueName: \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\") pod \"test-curl-7024-d75da0\" (UID: \"9bb252c6-c92b-495e-a55e-4e996fe133f8\") " pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:31.836140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:31.836112    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdq7\" (UniqueName: \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\") pod \"test-curl-7024-d75da0\" (UID: \"9bb252c6-c92b-495e-a55e-4e996fe133f8\") " pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:32.029803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.029761    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:32.212691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.212660    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:32.229563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.229143    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:32.472178 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:32.472143    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb252c6_c92b_495e_a55e_4e996fe133f8.slice/crio-15a3d0d927974ccd151fbec905f53b1d2fe5fc5f15a3ed339894f04dcb75af75 WatchSource:0}: Error finding container 15a3d0d927974ccd151fbec905f53b1d2fe5fc5f15a3ed339894f04dcb75af75: Status 404 returned error can't find the container with id 15a3d0d927974ccd151fbec905f53b1d2fe5fc5f15a3ed339894f04dcb75af75
Nov 03 20:25:32.476195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.476163    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-d75da0"]
Nov 03 20:25:32.611821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.611786    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-d75da0" event={"ID":"9bb252c6-c92b-495e-a55e-4e996fe133f8","Type":"ContainerStarted","Data":"99f0755fc1c6dcd8cff88b262e0871e59633a3dd5212470c12f93b9b2ee4e463"}
Nov 03 20:25:32.611821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.611835    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-d75da0" event={"ID":"9bb252c6-c92b-495e-a55e-4e996fe133f8","Type":"ContainerStarted","Data":"15a3d0d927974ccd151fbec905f53b1d2fe5fc5f15a3ed339894f04dcb75af75"}
Nov 03 20:25:32.612348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.611946    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-2c8921" podUID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" containerName="test-curl-7024-2c8921" containerID="cri-o://aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619" gracePeriod=2
Nov 03 20:25:32.612348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.612277    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:32.613319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.613293    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:25:32.663409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:32.663359    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-d75da0" podStartSLOduration=1.663346016 podStartE2EDuration="1.663346016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:32.657643267 +0000 UTC m=+1274.866659264" watchObservedRunningTime="2026-11-03 20:25:32.663346016 +0000 UTC m=+1274.872362012"
Nov 03 20:25:33.790567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.774069    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-d75da0"]
Nov 03 20:25:33.853673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.840823    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-d75da0"]
Nov 03 20:25:33.853673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.845187    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-2d1637"]
Nov 03 20:25:33.853673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.846240    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9bb252c6-c92b-495e-a55e-4e996fe133f8" containerName="test-curl-7024-d75da0"
Nov 03 20:25:33.853673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.846260    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb252c6-c92b-495e-a55e-4e996fe133f8" containerName="test-curl-7024-d75da0"
Nov 03 20:25:33.853673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.846454    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb252c6-c92b-495e-a55e-4e996fe133f8" containerName="test-curl-7024-d75da0"
Nov 03 20:25:33.855844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:33.855821    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:34.011773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.011737    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-2d1637"]
Nov 03 20:25:34.025049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.025013    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:25:34.040350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.040315    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\") pod \"test-curl-7024-2d1637\" (UID: \"833e248a-2f74-49ab-a55e-205a1422e107\") " pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:34.048029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.047964    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:25:34.048360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.048283    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.079321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.075154    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-eb87f429\"/\"default-dockercfg-qwn56\""
Nov 03 20:25:34.087053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.087028    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-eb87f429\"/\"openshift-service-ca.crt\""
Nov 03 20:25:34.087691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.087669    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-eb87f429\"/\"e2e-lifecycle-model-eb87f429-kserve-self-signed-certs\""
Nov 03 20:25:34.087919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.087906    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-eb87f429\"/\"kube-root-ca.crt\""
Nov 03 20:25:34.281438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.281390    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltg7\" (UniqueName: \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.281841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.281814    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\") pod \"test-curl-7024-2d1637\" (UID: \"833e248a-2f74-49ab-a55e-205a1422e107\") " pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:34.282012 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.281993    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.282213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.282197    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.282405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.282388    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.282572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.282554    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.282713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.282697    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.402721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.402618    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltg7\" (UniqueName: \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.403064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.403036    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.403295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.403276    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.403492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.403474    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.403671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.403652    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.403804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.403786    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.405005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.404982    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.405834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.405771    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.405985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.405876    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.407073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.407049    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\") pod \"test-curl-7024-2d1637\" (UID: \"833e248a-2f74-49ab-a55e-205a1422e107\") " pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:34.438355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.438324    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.469798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.469727    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltg7\" (UniqueName: \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.469798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.469787    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\") pod \"e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") " pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.568561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.567779    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:34.597231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.595861    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:34.760580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:34.757302    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-d75da0" podUID="9bb252c6-c92b-495e-a55e-4e996fe133f8" containerName="test-curl-7024-d75da0" containerID="cri-o://99f0755fc1c6dcd8cff88b262e0871e59633a3dd5212470c12f93b9b2ee4e463" gracePeriod=2
Nov 03 20:25:35.921914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:35.897145    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-2d1637"]
Nov 03 20:25:35.977887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:35.977858    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:35.985120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:35.985062    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:25:35.996610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:35.995218    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\" (UniqueName: \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\") pod \"e7d2ad80-ebd3-462c-b473-ad94d5696fce\" (UID: \"e7d2ad80-ebd3-462c-b473-ad94d5696fce\") "
Nov 03 20:25:36.008049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.007382    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-2d1637" event={"ID":"833e248a-2f74-49ab-a55e-205a1422e107","Type":"ContainerStarted","Data":"022692c0da6563c4a721dbf7cddb9be24829df562ac3af0866c75d21acd0e65c"}
Nov 03 20:25:36.010054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.009513    5424 generic.go:356] "Generic (PLEG): container finished" podID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" containerID="aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619" exitCode=137
Nov 03 20:25:36.010054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.009621    5424 scope.go:122] "RemoveContainer" containerID="aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619"
Nov 03 20:25:36.010054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.009793    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2c8921"
Nov 03 20:25:36.013984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.013908    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" event={"ID":"85dd4a05-42d1-4b10-b649-293b1795ffa7","Type":"ContainerStarted","Data":"380f3b13c080f41fc690c2d4cbfd42ef60e963659144bc27d3f65fd9f9cb1b10"}
Nov 03 20:25:36.021668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.021629    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx" pod "e7d2ad80-ebd3-462c-b473-ad94d5696fce" (UID: "e7d2ad80-ebd3-462c-b473-ad94d5696fce"). InnerVolumeSpecName "kube-api-access-tqxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:36.076302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.072052    5424 scope.go:122] "RemoveContainer" containerID="aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619"
Nov 03 20:25:36.076302 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:36.072385    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619\": container with ID starting with aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619 not found: ID does not exist" containerID="aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619"
Nov 03 20:25:36.076302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.072418    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619"} err="failed to get container status \"aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619\": rpc error: code = NotFound desc = could not find container \"aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619\": container with ID starting with aaa05687862cff8d6e6b03d78a74ff79520c4deaa384a6d458b92b3e6264c619 not found: ID does not exist"
Nov 03 20:25:36.101220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:36.101191    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxqx\" (UniqueName: \"kubernetes.io/projected/e7d2ad80-ebd3-462c-b473-ad94d5696fce-kube-api-access-tqxqx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:37.119652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.110733    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-2d1637" event={"ID":"833e248a-2f74-49ab-a55e-205a1422e107","Type":"ContainerStarted","Data":"c3a4095f6eccbaff6e501382e471b3a04c0fa9a66ffe6b07a5f284e0be5d2a95"}
Nov 03 20:25:37.119652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.115628    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" event={"ID":"85dd4a05-42d1-4b10-b649-293b1795ffa7","Type":"ContainerStarted","Data":"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"}
Nov 03 20:25:37.119652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.117920    5424 generic.go:356] "Generic (PLEG): container finished" podID="9bb252c6-c92b-495e-a55e-4e996fe133f8" containerID="99f0755fc1c6dcd8cff88b262e0871e59633a3dd5212470c12f93b9b2ee4e463" exitCode=137
Nov 03 20:25:37.198373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.198344    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:37.201184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.200997    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-2d1637" podStartSLOduration=4.200978902 podStartE2EDuration="4.200978902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:37.151118124 +0000 UTC m=+1279.360134121" watchObservedRunningTime="2026-11-03 20:25:37.200978902 +0000 UTC m=+1279.409994901"
Nov 03 20:25:37.337313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.337273    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\" (UniqueName: \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\") pod \"9bb252c6-c92b-495e-a55e-4e996fe133f8\" (UID: \"9bb252c6-c92b-495e-a55e-4e996fe133f8\") "
Nov 03 20:25:37.350590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.350505    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7" pod "9bb252c6-c92b-495e-a55e-4e996fe133f8" (UID: "9bb252c6-c92b-495e-a55e-4e996fe133f8"). InnerVolumeSpecName "kube-api-access-hgdq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:37.442333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.442295    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdq7\" (UniqueName: \"kubernetes.io/projected/9bb252c6-c92b-495e-a55e-4e996fe133f8-kube-api-access-hgdq7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:37.597522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.597472    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:37.925365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.925331    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb252c6-c92b-495e-a55e-4e996fe133f8" path="/var/lib/kubelet/pods/9bb252c6-c92b-495e-a55e-4e996fe133f8/volumes"
Nov 03 20:25:37.925862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.925846    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d2ad80-ebd3-462c-b473-ad94d5696fce" path="/var/lib/kubelet/pods/e7d2ad80-ebd3-462c-b473-ad94d5696fce/volumes"
Nov 03 20:25:37.996194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.996096    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" podStartSLOduration=4.996076049 podStartE2EDuration="4.996076049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:37.199693765 +0000 UTC m=+1279.408709778" watchObservedRunningTime="2026-11-03 20:25:37.996076049 +0000 UTC m=+1280.205092047"
Nov 03 20:25:37.997964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:37.997926    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-2d1637"]
Nov 03 20:25:38.003716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.003689    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-2d1637"]
Nov 03 20:25:38.128955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.128927    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-d75da0"
Nov 03 20:25:38.129414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.128989    5424 scope.go:122] "RemoveContainer" containerID="99f0755fc1c6dcd8cff88b262e0871e59633a3dd5212470c12f93b9b2ee4e463"
Nov 03 20:25:38.153334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.153301    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7024-f06a05"]
Nov 03 20:25:38.154339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.154313    5424 cpu_manager.go:414] "containerMap: removing container" podUID="833e248a-2f74-49ab-a55e-205a1422e107" containerName="test-curl-7024-2d1637"
Nov 03 20:25:38.154504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.154341    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833e248a-2f74-49ab-a55e-205a1422e107" containerName="test-curl-7024-2d1637"
Nov 03 20:25:38.154633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.154613    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="833e248a-2f74-49ab-a55e-205a1422e107" containerName="test-curl-7024-2d1637"
Nov 03 20:25:38.159228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.159200    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:38.173670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.173350    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-f06a05"]
Nov 03 20:25:38.263981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.263947    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjsd\" (UniqueName: \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\") pod \"test-curl-7024-f06a05\" (UID: \"72b320b7-cf22-455e-8501-1856ef089a97\") " pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:38.287631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.287603    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:25:38.291907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.291881    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.310626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.310599    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:25:38.311146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.311122    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-55b09bd8\""
Nov 03 20:25:38.365707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.365675    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:25:38.365897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.365714    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9ll\" (UniqueName: \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.365897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.365789    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjsd\" (UniqueName: \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\") pod \"test-curl-7024-f06a05\" (UID: \"72b320b7-cf22-455e-8501-1856ef089a97\") " pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:38.366019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.365956    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.369249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.369227    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.386402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.386372    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-55b09bd8-dockercfg-dj6lj\""
Nov 03 20:25:38.388545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.388498    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:25:38.438558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.430322    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjsd\" (UniqueName: \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\") pod \"test-curl-7024-f06a05\" (UID: \"72b320b7-cf22-455e-8501-1856ef089a97\") " pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:38.474128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.473966    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.474348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.474292    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9ll\" (UniqueName: \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.475639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.475608    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.503126 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.502823    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:38.528300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.509620    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9ll\" (UniqueName: \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\") pod \"payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.614571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.600291    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5nr\" (UniqueName: \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.614571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.600595    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.622692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.622657    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:38.739903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.706432    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5nr\" (UniqueName: \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.739903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.708150    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.739903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.719165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.739903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.733369    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5nr\" (UniqueName: \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\") pod \"payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") " pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:38.988887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:38.988856    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:39.099649 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:39.099590    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72b320b7_cf22_455e_8501_1856ef089a97.slice/crio-6c94f04ba1a9a8f51116adc902c6a69265ecd9940f8fe54e6e0af749f64eb630 WatchSource:0}: Error finding container 6c94f04ba1a9a8f51116adc902c6a69265ecd9940f8fe54e6e0af749f64eb630: Status 404 returned error can't find the container with id 6c94f04ba1a9a8f51116adc902c6a69265ecd9940f8fe54e6e0af749f64eb630
Nov 03 20:25:39.104067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:39.104039    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7024-f06a05"]
Nov 03 20:25:39.147284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:39.147246    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:25:39.172298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:39.172264    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-f06a05" event={"ID":"72b320b7-cf22-455e-8501-1856ef089a97","Type":"ContainerStarted","Data":"6c94f04ba1a9a8f51116adc902c6a69265ecd9940f8fe54e6e0af749f64eb630"}
Nov 03 20:25:39.174925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:39.174898    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-2d1637" podUID="833e248a-2f74-49ab-a55e-205a1422e107" containerName="test-curl-7024-2d1637" containerID="cri-o://c3a4095f6eccbaff6e501382e471b3a04c0fa9a66ffe6b07a5f284e0be5d2a95" gracePeriod=2
Nov 03 20:25:39.188414 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:39.188384    5424 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7024-2d1637\" is forbidden: User \"system:node:ip-10-0-136-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-244.ec2.internal' and this object" podUID="833e248a-2f74-49ab-a55e-205a1422e107" pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:39.303193 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:39.303155    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda135bcb_3008_4a4f_a77b_34194d81446c.slice/crio-cee6acd537db0e5c6594707a14224783eb553ce4d7a8f944040be8e972715234 WatchSource:0}: Error finding container cee6acd537db0e5c6594707a14224783eb553ce4d7a8f944040be8e972715234: Status 404 returned error can't find the container with id cee6acd537db0e5c6594707a14224783eb553ce4d7a8f944040be8e972715234
Nov 03 20:25:39.304278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:39.304244    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:25:40.184441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.184400    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" event={"ID":"c4b20869-0ef7-45bf-a431-70b89761a9f4","Type":"ContainerStarted","Data":"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"}
Nov 03 20:25:40.184910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.184461    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" event={"ID":"c4b20869-0ef7-45bf-a431-70b89761a9f4","Type":"ContainerStarted","Data":"defaba414a790a7a69b95fa518ef42594217584e4072cf1134fd7e8663fa2d77"}
Nov 03 20:25:40.184910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.184489    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:40.186275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.186240    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7024-f06a05" event={"ID":"72b320b7-cf22-455e-8501-1856ef089a97","Type":"ContainerStarted","Data":"d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc"}
Nov 03 20:25:40.187858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.187833    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" event={"ID":"da135bcb-3008-4a4f-a77b-34194d81446c","Type":"ContainerStarted","Data":"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"}
Nov 03 20:25:40.188009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.187879    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" event={"ID":"da135bcb-3008-4a4f-a77b-34194d81446c","Type":"ContainerStarted","Data":"cee6acd537db0e5c6594707a14224783eb553ce4d7a8f944040be8e972715234"}
Nov 03 20:25:40.188009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.187918    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:40.226957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.226911    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" podStartSLOduration=2.226898518 podStartE2EDuration="2.226898518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:40.224579307 +0000 UTC m=+1282.433595301" watchObservedRunningTime="2026-11-03 20:25:40.226898518 +0000 UTC m=+1282.435914514"
Nov 03 20:25:40.264714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.264669    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7024-f06a05" podStartSLOduration=2.264657638 podStartE2EDuration="2.264657638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:40.261494552 +0000 UTC m=+1282.470510543" watchObservedRunningTime="2026-11-03 20:25:40.264657638 +0000 UTC m=+1282.473673636"
Nov 03 20:25:40.303977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:40.303934    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" podStartSLOduration=2.303922291 podStartE2EDuration="2.303922291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:40.301621413 +0000 UTC m=+1282.510637411" watchObservedRunningTime="2026-11-03 20:25:40.303922291 +0000 UTC m=+1282.512938287"
Nov 03 20:25:41.073551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.072377    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7024-f06a05"]
Nov 03 20:25:41.090085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.090056    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7024-f06a05"]
Nov 03 20:25:41.203632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.203594    5424 generic.go:356] "Generic (PLEG): container finished" podID="833e248a-2f74-49ab-a55e-205a1422e107" containerID="c3a4095f6eccbaff6e501382e471b3a04c0fa9a66ffe6b07a5f284e0be5d2a95" exitCode=137
Nov 03 20:25:41.378417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.378389    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:41.456215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.456177    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\") pod \"833e248a-2f74-49ab-a55e-205a1422e107\" (UID: \"833e248a-2f74-49ab-a55e-205a1422e107\") "
Nov 03 20:25:41.458437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.458407    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx" pod "833e248a-2f74-49ab-a55e-205a1422e107" (UID: "833e248a-2f74-49ab-a55e-205a1422e107"). InnerVolumeSpecName "kube-api-access-ng2bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.559073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.559041    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2bx\" (UniqueName: \"kubernetes.io/projected/833e248a-2f74-49ab-a55e-205a1422e107-kube-api-access-ng2bx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:41.924238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:41.924205    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833e248a-2f74-49ab-a55e-205a1422e107" path="/var/lib/kubelet/pods/833e248a-2f74-49ab-a55e-205a1422e107/volumes"
Nov 03 20:25:42.360065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.349494    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:25:42.360065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.354946    5424 cpu_manager.go:414] "containerMap: removing container" podUID="72b320b7-cf22-455e-8501-1856ef089a97" containerName="test-curl-7024-f06a05"
Nov 03 20:25:42.360065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.354970    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b320b7-cf22-455e-8501-1856ef089a97" containerName="test-curl-7024-f06a05"
Nov 03 20:25:42.360065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.355123    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b320b7-cf22-455e-8501-1856ef089a97" containerName="test-curl-7024-f06a05"
Nov 03 20:25:42.401558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.376157    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-2d1637"
Nov 03 20:25:42.401558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.377286    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7024-f06a05" podUID="72b320b7-cf22-455e-8501-1856ef089a97" containerName="test-curl-7024-f06a05" containerID="cri-o://d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc" gracePeriod=2
Nov 03 20:25:42.401558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.384714    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:25:42.401558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.384792    5424 scope.go:122] "RemoveContainer" containerID="c3a4095f6eccbaff6e501382e471b3a04c0fa9a66ffe6b07a5f284e0be5d2a95"
Nov 03 20:25:42.401558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.385020    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.461508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.461234    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dc7b94-openshift-default-dockercfg-54ddw\""
Nov 03 20:25:42.473565 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:42.470688    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:25:42.506875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.506830    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.506991    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507053    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507110    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507227    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507274    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6x8\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507315    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507365    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.507496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.507426    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609268    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609325    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609365    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609444    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609490    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6x8\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609547    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609600    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609680    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.609972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.609844    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.610289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.610240    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.610767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.610745    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.611933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.611908    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.612738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.612711    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.612850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.612833    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.617586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.614773    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.618096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.618073    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.633583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.633418    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.637341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.637311    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6x8\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\") pod \"e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") " pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:42.730553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:42.728282    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:43.108692 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:43.108662    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284ab405_63f0_43b3_91b7_c435b321481e.slice/crio-cbc6d751adca9f6a6a543921b02fa6aef9ac4f012238ff2d0cd5163e35f5e2b4 WatchSource:0}: Error finding container cbc6d751adca9f6a6a543921b02fa6aef9ac4f012238ff2d0cd5163e35f5e2b4: Status 404 returned error can't find the container with id cbc6d751adca9f6a6a543921b02fa6aef9ac4f012238ff2d0cd5163e35f5e2b4
Nov 03 20:25:43.111594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.111569    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:25:43.111838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.111809    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:43.111888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.111875    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:43.111937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.111925    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:43.392000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.391895    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerStarted","Data":"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"}
Nov 03 20:25:43.392000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.391963    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerStarted","Data":"cbc6d751adca9f6a6a543921b02fa6aef9ac4f012238ff2d0cd5163e35f5e2b4"}
Nov 03 20:25:43.435419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.435366    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podStartSLOduration=1.43535278 podStartE2EDuration="1.43535278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:43.432443359 +0000 UTC m=+1285.641459358" watchObservedRunningTime="2026-11-03 20:25:43.43535278 +0000 UTC m=+1285.644368777"
Nov 03 20:25:43.729258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:43.729174    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:44.599833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.599607    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:44.630552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.630430    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:44.735717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.735687    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:44.739300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.739273    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:44.854840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.854810    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\" (UniqueName: \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\") pod \"72b320b7-cf22-455e-8501-1856ef089a97\" (UID: \"72b320b7-cf22-455e-8501-1856ef089a97\") "
Nov 03 20:25:44.857016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.856989    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd" pod "72b320b7-cf22-455e-8501-1856ef089a97" (UID: "72b320b7-cf22-455e-8501-1856ef089a97"). InnerVolumeSpecName "kube-api-access-lfjsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:44.957233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:44.957151    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjsd\" (UniqueName: \"kubernetes.io/projected/72b320b7-cf22-455e-8501-1856ef089a97-kube-api-access-lfjsd\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:25:45.416821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.416780    5424 generic.go:356] "Generic (PLEG): container finished" podID="72b320b7-cf22-455e-8501-1856ef089a97" containerID="d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc" exitCode=137
Nov 03 20:25:45.417005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.416827    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:45.417005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.416873    5424 scope.go:122] "RemoveContainer" containerID="d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc"
Nov 03 20:25:45.418785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.418269    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:45.419215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.419187    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:25:45.430341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.430144    5424 scope.go:122] "RemoveContainer" containerID="d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc"
Nov 03 20:25:45.430502 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:45.430484    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc\": container with ID starting with d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc not found: ID does not exist" containerID="d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc"
Nov 03 20:25:45.430623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.430515    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc"} err="failed to get container status \"d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc\": rpc error: code = NotFound desc = could not find container \"d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc\": container with ID starting with d75a9ad19dcebed859657956eb7fb54ac9e2237ee2fcefac495515bd1b3b09bc not found: ID does not exist"
Nov 03 20:25:45.431960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.431939    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:25:45.475335 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:25:45.475295    5424 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7024-f06a05\" is forbidden: User \"system:node:ip-10-0-136-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-244.ec2.internal' and this object" podUID="72b320b7-cf22-455e-8501-1856ef089a97" pod="opendatahub/test-curl-7024-f06a05"
Nov 03 20:25:45.923375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:45.923348    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b320b7-cf22-455e-8501-1856ef089a97" path="/var/lib/kubelet/pods/72b320b7-cf22-455e-8501-1856ef089a97/volumes"
Nov 03 20:25:48.086718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.086687    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:25:48.092058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.092035    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.112581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.105695    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:25:48.112581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.111649    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dc7b94\""
Nov 03 20:25:48.147118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.147063    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dc7b94-dockercfg-lgtq5\""
Nov 03 20:25:48.244047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.243308    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.244047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.243495    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsb7\" (UniqueName: \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.345970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.345833    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.346172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.346040    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsb7\" (UniqueName: \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.346766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.346741    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.367303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.367266    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsb7\" (UniqueName: \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\") pod \"payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") " pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.408522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.408488    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:48.737411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:48.737382    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:25:49.463976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:49.463935    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" event={"ID":"3f16513b-6b89-4450-9760-259709a68873","Type":"ContainerStarted","Data":"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"}
Nov 03 20:25:49.464396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:49.464004    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" event={"ID":"3f16513b-6b89-4450-9760-259709a68873","Type":"ContainerStarted","Data":"6d02f9a8736150a10f1fdfad4a06d589a503aaaa3682d3b2b44ee4fa25af7a7e"}
Nov 03 20:25:49.464396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:49.464033    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:25:49.506268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:49.506214    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" podStartSLOduration=1.506198674 podStartE2EDuration="1.506198674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:49.501211338 +0000 UTC m=+1291.710227335" watchObservedRunningTime="2026-11-03 20:25:49.506198674 +0000 UTC m=+1291.715214679"
Nov 03 20:25:50.580331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.580299    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:25:50.583941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.583920    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.604118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.604084    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:25:50.609405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.609361    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-76b894-dockercfg-sl562\""
Nov 03 20:25:50.609569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.609447    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-76b894\""
Nov 03 20:25:50.692435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.686584    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mm9v\" (UniqueName: \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.692435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.686877    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.796614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.788271    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mm9v\" (UniqueName: \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.796614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.788609    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.796614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.790281    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.813057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.813025    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mm9v\" (UniqueName: \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\") pod \"payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.895954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.886688    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:25:50.926646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.926606    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:50.928685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.928663    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:50.954410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:50.954375    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:25:51.097306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.097072    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.097306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.097253    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hxz\" (UniqueName: \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.199071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.198966    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.199236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.199083    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hxz\" (UniqueName: \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.199699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.199672    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.206973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.206923    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:25:51.207323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.207303    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:25:51.219649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.219614    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hxz\" (UniqueName: \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\") pod \"payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.271599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.270087    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:51.274853 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:51.274820    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd295797f_5a82_4283_a555_fe49bbdf4947.slice/crio-b3fe00ecff8545875ca0619581cefda96850b3bbb6dad76d75760831046d5409 WatchSource:0}: Error finding container b3fe00ecff8545875ca0619581cefda96850b3bbb6dad76d75760831046d5409: Status 404 returned error can't find the container with id b3fe00ecff8545875ca0619581cefda96850b3bbb6dad76d75760831046d5409
Nov 03 20:25:51.277836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.277810    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:25:51.487026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.486970    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" event={"ID":"d295797f-5a82-4283-a555-fe49bbdf4947","Type":"ContainerStarted","Data":"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"}
Nov 03 20:25:51.487223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.487048    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" event={"ID":"d295797f-5a82-4283-a555-fe49bbdf4947","Type":"ContainerStarted","Data":"b3fe00ecff8545875ca0619581cefda96850b3bbb6dad76d75760831046d5409"}
Nov 03 20:25:51.487223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.487080    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:25:51.526862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.526813    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" podStartSLOduration=1.526795793 podStartE2EDuration="1.526795793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:51.524514555 +0000 UTC m=+1293.733530553" watchObservedRunningTime="2026-11-03 20:25:51.526795793 +0000 UTC m=+1293.735811787"
Nov 03 20:25:51.617729 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:51.617686    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9297e4fc_238d_43a5_b646_9c8688c55035.slice/crio-fe66f10fcd68e91be3d4862d65f2a81b8155f2304d8f84bb5f3501a53bf654dc WatchSource:0}: Error finding container fe66f10fcd68e91be3d4862d65f2a81b8155f2304d8f84bb5f3501a53bf654dc: Status 404 returned error can't find the container with id fe66f10fcd68e91be3d4862d65f2a81b8155f2304d8f84bb5f3501a53bf654dc
Nov 03 20:25:51.620068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:51.620048    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:25:52.497419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.497379    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" event={"ID":"9297e4fc-238d-43a5-b646-9c8688c55035","Type":"ContainerStarted","Data":"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"}
Nov 03 20:25:52.497617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.497437    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" event={"ID":"9297e4fc-238d-43a5-b646-9c8688c55035","Type":"ContainerStarted","Data":"fe66f10fcd68e91be3d4862d65f2a81b8155f2304d8f84bb5f3501a53bf654dc"}
Nov 03 20:25:52.497617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.497468    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:25:52.538360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.538299    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" podStartSLOduration=2.53828393 podStartE2EDuration="2.53828393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:52.535752292 +0000 UTC m=+1294.744768288" watchObservedRunningTime="2026-11-03 20:25:52.53828393 +0000 UTC m=+1294.747299940"
Nov 03 20:25:52.553317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.553286    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:25:52.557336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.557316    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.573762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.573718    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eb87f429-openshift-default-dockercfg-fwjlv\""
Nov 03 20:25:52.577110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.577086    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:25:52.626933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.626898    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627017    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627091    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627203    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627315    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627395    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627443    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mq7x\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627483    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.627656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.627507    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.728924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.728888    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729103 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.728994    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729103 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729045    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mq7x\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729103 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729085    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729113    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729239    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729266    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729339    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729376    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729404    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729469    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729592    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.729710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.729599    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.730203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.730180    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.731727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.731701    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.732066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.732049    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.750973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.750898    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.752101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.752048    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mq7x\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:52.873762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:52.873728    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:53.031841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.031749    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"]
Nov 03 20:25:53.039500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.039467    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6247b024-hxhlc"]
Nov 03 20:25:53.194205 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:53.194173    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f73b3c_a68a_4433_9ee0_e5f67f7edff6.slice/crio-e9370851670c12f351ecca029c5fadbb971a14e9704cb47fa4243af138b14b9d WatchSource:0}: Error finding container e9370851670c12f351ecca029c5fadbb971a14e9704cb47fa4243af138b14b9d: Status 404 returned error can't find the container with id e9370851670c12f351ecca029c5fadbb971a14e9704cb47fa4243af138b14b9d
Nov 03 20:25:53.196897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.196869    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:25:53.197385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.197350    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:53.197439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.197419    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:53.197475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.197449    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:53.511757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.511721    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" event={"ID":"86f73b3c-a68a-4433-9ee0-e5f67f7edff6","Type":"ContainerStarted","Data":"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"}
Nov 03 20:25:53.511993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.511973    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" event={"ID":"86f73b3c-a68a-4433-9ee0-e5f67f7edff6","Type":"ContainerStarted","Data":"e9370851670c12f351ecca029c5fadbb971a14e9704cb47fa4243af138b14b9d"}
Nov 03 20:25:53.554856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.554738    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podStartSLOduration=1.554718144 podStartE2EDuration="1.554718144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:53.551945158 +0000 UTC m=+1295.760961154" watchObservedRunningTime="2026-11-03 20:25:53.554718144 +0000 UTC m=+1295.763734141"
Nov 03 20:25:53.874142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.874064    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:25:53.921947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:53.921917    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34cfe63f-7002-493f-8c6a-a613b20d46fe" path="/var/lib/kubelet/pods/34cfe63f-7002-493f-8c6a-a613b20d46fe/volumes"
Nov 03 20:25:54.029233 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:54.029201    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"]
Nov 03 20:25:54.034176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:54.034147    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e0cf92d-zmjp5"]
Nov 03 20:25:54.875515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:54.875482    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:54.875954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:54.875584    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:25:55.874919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:55.874884    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:55.875107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:55.874931    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:25:55.922632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:55.922592    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146e2043-f1b3-4135-87ce-be4e86a01cf6" path="/var/lib/kubelet/pods/146e2043-f1b3-4135-87ce-be4e86a01cf6/volumes"
Nov 03 20:25:56.874273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:56.874239    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:56.874428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:56.874290    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:25:57.048127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.048096    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:25:57.052479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.052456    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.077883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.077853    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:25:57.083855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.083829    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-e3ba87-openshift-default-dockercfg-gblhq\""
Nov 03 20:25:57.145850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.145737    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.145885    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vns7m\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.145940    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.145988    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.146104    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.146158    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.146247    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.146308    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.146541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.146505    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248394    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248508    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248618    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vns7m\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248681    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248738    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.248837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248816    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248869    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248877    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248868    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.248959    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.249025    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.249168    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.249477    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.249617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.249572    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.253472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.253444    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.253792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.253751    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.268902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.268872    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.269038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.268883    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vns7m\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.367277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.367243    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:57.721926 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:25:57.719925    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda948690_b1a6_4cb1_b51e_c9456d44e39b.slice/crio-5b748813d7e38391df5b0e6454d5dadf0f7f4819be64f4c10cc76a71ed83436f WatchSource:0}: Error finding container 5b748813d7e38391df5b0e6454d5dadf0f7f4819be64f4c10cc76a71ed83436f: Status 404 returned error can't find the container with id 5b748813d7e38391df5b0e6454d5dadf0f7f4819be64f4c10cc76a71ed83436f
Nov 03 20:25:57.738724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.738689    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:25:57.740714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.740668    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:57.740837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.740765    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:57.740837 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.740809    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:57.876632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.874319    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:57.876632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:57.874385    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:25:58.560779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:58.560745    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" event={"ID":"da948690-b1a6-4cb1-b51e-c9456d44e39b","Type":"ContainerStarted","Data":"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"}
Nov 03 20:25:58.561258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:58.560812    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" event={"ID":"da948690-b1a6-4cb1-b51e-c9456d44e39b","Type":"ContainerStarted","Data":"5b748813d7e38391df5b0e6454d5dadf0f7f4819be64f4c10cc76a71ed83436f"}
Nov 03 20:25:58.615440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:58.615386    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podStartSLOduration=1.615371298 podStartE2EDuration="1.615371298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:58.614107173 +0000 UTC m=+1300.823123172" watchObservedRunningTime="2026-11-03 20:25:58.615371298 +0000 UTC m=+1300.824387294"
Nov 03 20:25:58.874401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:58.874314    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:58.874401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:58.874369    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:25:59.367991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:59.367958    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:25:59.369999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:59.369973    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:25:59.370125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:59.370021    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:25:59.874800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:59.874756    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:25:59.875218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:25:59.874815    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:00.031653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.031621    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"]
Nov 03 20:26:00.043074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.043043    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed8e6f-rnwp9"]
Nov 03 20:26:00.368406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.368375    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:00.368622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.368426    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:00.476630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.476596    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:26:00.874150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.874106    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:00.874369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:00.874164    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:01.368485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:01.368451    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:01.368898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:01.368513    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:01.874638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:01.874605    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:01.874818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:01.874661    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:01.922252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:01.922215    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54b8845-c35f-43fe-893e-486bb7b28ad2" path="/var/lib/kubelet/pods/f54b8845-c35f-43fe-893e-486bb7b28ad2/volumes"
Nov 03 20:26:02.368484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:02.368442    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:02.368713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:02.368509    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:02.501545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:02.501501    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:26:02.874017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:02.873986    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:02.874192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:02.874036    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:03.368548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:03.368499    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:03.368744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:03.368568    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:03.516306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:03.516279    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:26:03.874959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:03.874920    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:03.875157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:03.874978    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:04.030354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.030315    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"]
Nov 03 20:26:04.035431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.035401    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21caae8b-s2kvx"]
Nov 03 20:26:04.368354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.368318    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:04.368555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.368383    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:04.874815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.874781    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:04.875209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:04.874837    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:05.368600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:05.368562    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:05.368782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:05.368632    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:05.874833 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:05.874801    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:05.875380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:05.874861    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:05.922127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:05.922092    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b016d30e-2d15-4de3-a6dd-180c057a717a" path="/var/lib/kubelet/pods/b016d30e-2d15-4de3-a6dd-180c057a717a/volumes"
Nov 03 20:26:06.371706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:06.371671    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:06.372022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:06.371997    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:06.878082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:06.878044    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:06.878720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:06.878688    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:07.065418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.065385    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"]
Nov 03 20:26:07.073375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.073328    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4307b6-k7tbf"]
Nov 03 20:26:07.369054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.368999    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:07.369283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.369071    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:07.882999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.882960    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:07.883613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.883579    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:07.884080 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.884062    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:26:07.909501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.909462    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:26:07.965748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.952638    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.989649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.989603    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.989855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.989779    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.989927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.989864    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.989978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.989928    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.990026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.990010    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.990128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.990099    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.990185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.990126    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.990185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.990149    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:07.990185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:07.990168    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcdp\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.003045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.003013    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc45015a-8ca6-488c-9ef1-ea4c0af7e750" path="/var/lib/kubelet/pods/dc45015a-8ca6-488c-9ef1-ea4c0af7e750/volumes"
Nov 03 20:26:08.003389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.003375    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:26:08.091355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091321    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091398    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091458    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091510    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091576    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091638    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091673    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091705    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.091885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.091736    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcdp\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.092231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.092202    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.092495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.092474    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.092694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.092668    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.092803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.092780    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.093050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.093029    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.095632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.095039    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.095632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.095323    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.114515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.114476    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.115441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.115418    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcdp\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.303577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.303544    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:08.368556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.368250    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:08.368556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.368328    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:08.716697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.716666    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:26:08.752788 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:08.751676    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc8264e_3e1b_452b_ad1d_fe7527c85cac.slice/crio-cd445ec295db5c1eaa20ea86cc5f20a0a113ce789ff438298766a317a0e0c56f WatchSource:0}: Error finding container cd445ec295db5c1eaa20ea86cc5f20a0a113ce789ff438298766a317a0e0c56f: Status 404 returned error can't find the container with id cd445ec295db5c1eaa20ea86cc5f20a0a113ce789ff438298766a317a0e0c56f
Nov 03 20:26:08.774430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.774387    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:08.774570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.774468    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:08.774570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.774508    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:08.875621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.875243    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:08.875621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:08.875304    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:09.374716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.370586    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:09.374716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.370646    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:09.800686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.799588    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" event={"ID":"4fc8264e-3e1b-452b-ad1d-fe7527c85cac","Type":"ContainerStarted","Data":"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"}
Nov 03 20:26:09.800686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.799687    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" event={"ID":"4fc8264e-3e1b-452b-ad1d-fe7527c85cac","Type":"ContainerStarted","Data":"cd445ec295db5c1eaa20ea86cc5f20a0a113ce789ff438298766a317a0e0c56f"}
Nov 03 20:26:09.882161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.879416    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:09.882161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.879484    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:09.892252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:09.892186    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podStartSLOduration=2.8921642690000002 podStartE2EDuration="2.892164269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:09.873092478 +0000 UTC m=+1312.082108474" watchObservedRunningTime="2026-11-03 20:26:09.892164269 +0000 UTC m=+1312.101180266"
Nov 03 20:26:10.304279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.304076    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:10.305364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.305326    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:10.305559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.305519    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:10.369555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.368959    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:10.369555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.369022    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:10.878554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.878089    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:10.878554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:10.878157    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:11.061557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.055584    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"]
Nov 03 20:26:11.100592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.092451    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e60e6ebf-z7wcs"]
Nov 03 20:26:11.308558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.308107    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:11.308558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.308163    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:11.369554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.368865    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:11.369554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.368926    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:11.889552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.885160    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:26:11.889552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.885214    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:26:11.993119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:11.988723    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b489a858-3f0a-4924-8602-5c1f2fd4f5f8" path="/var/lib/kubelet/pods/b489a858-3f0a-4924-8602-5c1f2fd4f5f8/volumes"
Nov 03 20:26:12.344582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.306245    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:12.344582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.306311    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:12.393653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.372662    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:26:12.478549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.417609    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:26:12.478549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.394079    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:26:12.494072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.483931    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:26:12.494072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.492132    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:26:12.499308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.499090    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.502660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.502485    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:26:12.502802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.502616    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.504639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.504608    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:26:12.564230 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:26:12.564193    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:26:12.585285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585255    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.585445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585310    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.585445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585411    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.585612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585584    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.585672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585642    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kh2\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.585783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585748    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.585928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585912    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.585994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585940    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.585994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.585979    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.586236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586209    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.586320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586303    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.586364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586348    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.586415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586386    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnmc\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.586561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586520    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.586626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586603    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.586685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586633    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.586685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586658    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.586685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.586677    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.688516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688430    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.688692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688520    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.688692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688616    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.688692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688645    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.688692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688676    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.688917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688761    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.688917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688813    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.688917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688866    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.689061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688957    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.689061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.688996    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kh2\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.689061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689043    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689511    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689500    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689524    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689646    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689710    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.689899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689854    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.689934    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.690433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690405    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690441    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690549 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690489    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.690675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690610    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690701    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690830    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.690937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.690915    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.691178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.691126    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.691178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.691166    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.691573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.691519    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.691654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.691607    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnmc\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.694143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.694120    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.694221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.694120    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.695181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.695163    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.711561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.711496    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnmc\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.711997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.711978    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.715140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.715118    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kh2\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.716647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.716627    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:12.862556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.862501    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:12.870579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:12.870347    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:13.151579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.148826    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:26:13.162351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.162323    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:26:13.259743 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:13.259710    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4c1c45_82d9_4f8d_b5d9_5be3cc09c03d.slice/crio-500b22e963fca0a0107b86dbead0e5f1d64d382dfee95d0cd07167f645620c87 WatchSource:0}: Error finding container 500b22e963fca0a0107b86dbead0e5f1d64d382dfee95d0cd07167f645620c87: Status 404 returned error can't find the container with id 500b22e963fca0a0107b86dbead0e5f1d64d382dfee95d0cd07167f645620c87
Nov 03 20:26:13.262382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.262334    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.262600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.262459    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.262600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.262569    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.262849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.262816    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:26:13.264447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.264409    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.264682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.264655    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.264850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.264822    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:13.280180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.280158    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:26:13.305916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.305863    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:13.309670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.305972    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:13.388877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.388849    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:26:13.390732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.390696    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:26:13.896462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.896429    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" event={"ID":"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d","Type":"ContainerStarted","Data":"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"}
Nov 03 20:26:13.896703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.896689    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" event={"ID":"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d","Type":"ContainerStarted","Data":"500b22e963fca0a0107b86dbead0e5f1d64d382dfee95d0cd07167f645620c87"}
Nov 03 20:26:13.899876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.899839    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" event={"ID":"02f07c45-83cc-41f9-bd4c-9b7364e58237","Type":"ContainerStarted","Data":"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"}
Nov 03 20:26:13.900010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.899897    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" event={"ID":"02f07c45-83cc-41f9-bd4c-9b7364e58237","Type":"ContainerStarted","Data":"30c769974a1036af53257915b990bdcc94fb66ce486f5a99b090ec8762c5815c"}
Nov 03 20:26:13.961234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.961171    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podStartSLOduration=1.9611563969999999 podStartE2EDuration="1.961156397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.942038925 +0000 UTC m=+1316.151054924" watchObservedRunningTime="2026-11-03 20:26:13.961156397 +0000 UTC m=+1316.170172392"
Nov 03 20:26:13.994721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:13.993789    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podStartSLOduration=1.993755964 podStartE2EDuration="1.993755964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.988434991 +0000 UTC m=+1316.197450989" watchObservedRunningTime="2026-11-03 20:26:13.993755964 +0000 UTC m=+1316.202771958"
Nov 03 20:26:14.304097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.304059    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.304489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.304108    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:14.870579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.867749    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:14.870579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.867870    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.870579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.867912    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:14.891488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.891256    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:14.891488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.891369    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.891488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:14.891419    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:15.304560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.304450    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.305220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.305189    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:15.330582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.316053    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:26:15.350588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.338410    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.392554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.391749    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hls\" (UniqueName: \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.392554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.392155    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.392554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.392218    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.392554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.392292    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.430567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.421593    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:26:15.499750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.495946    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:26:15.499750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.496772    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-49bxb\""
Nov 03 20:26:15.499750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.496882    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-76b894-serving-cert\""
Nov 03 20:26:15.499750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.496970    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:26:15.499750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.497029    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:26:15.510387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.510348    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.510717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.510682    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.511735 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.511595    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.511958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.511930    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.512516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.512497    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hls\" (UniqueName: \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.610377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.610297    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hls\" (UniqueName: \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.679552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.654195    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.753553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.730790    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 20:26:15.833562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.827005    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:15.844681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.844637    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.845226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.845198    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:26:15.846713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.846686    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.873187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.873099    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.873438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.873408    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:15.875780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.875758    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.875923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:15.875903    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:16.227730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.227639    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:16.306557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.306470    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 20:26:16.307117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.306574    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 20:26:16.628000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.627628    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.656559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.654789    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.751680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.750901    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.860634 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.840615    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.860940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.860900    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.861425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.861400    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.883554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.862057    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:26:16.883554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.873651    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:26:16.883554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.862765    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.905489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.904631    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.905489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.905114    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:26:16.925580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.913510    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:16.925580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.913585    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:16.925580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.913671    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:16.925580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.913697    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:16.941357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:16.940931    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:26:17.105213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.103630    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.122603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.110947    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:26:17.271390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.270720    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.419035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.418734    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.693149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.693067    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"]
Nov 03 20:26:17.720749 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.720708    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.809553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.796817    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:17.809553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.798695    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:17.880137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.841737    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.909284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.894807    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:26:18.092740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.912504    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.092740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.966863    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.092740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.082795    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:18.092740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:17.966920    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.092740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.082847    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:18.103907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.103868    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.150573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.145244    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:26:18.150573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.145311    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"]
Nov 03 20:26:18.150573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.145612    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:26:18.150573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.145659    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:26:18.226313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.226244    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:26:18.231474 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:26:18.229387    5424 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-fd1aa0db-sg9cs\" is forbidden: User \"system:node:ip-10-0-136-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-136-244.ec2.internal' and this object" podUID="e13b617e-38de-40e6-9f35-87629d33a97c" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1aa0db-sg9cs"
Nov 03 20:26:18.239361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.238939    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:26:18.247240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.247212    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" event={"ID":"210a4013-7945-4fd5-81e3-6c0168af8b38","Type":"ContainerStarted","Data":"85739f6066899952243b3e8d239bf4132ac77190d1ecf59e752065ac41ad2993"}
Nov 03 20:26:18.305864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.305837    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:26:18.867557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.866909    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.867557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.866975    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:18.876069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.875955    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.876069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:18.876012    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:19.797756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.797712    5424 scope.go:122] "RemoveContainer" containerID="ac2362c76ede00f915c872c0cca44d924db4c25fc195cfaf11971167e007e72d"
Nov 03 20:26:19.871789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.863694    5424 scope.go:122] "RemoveContainer" containerID="869073651d21f9a82a00fbc9d247eb6826a5563666b524ba1e56bca8436ce02c"
Nov 03 20:26:19.871789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.864166    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.871789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.864227    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 20:26:19.876974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.876922    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.877120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.876982    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:19.883560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.883501    5424 scope.go:122] "RemoveContainer" containerID="ca3e436af811f23510503b6983c9b4d540152e737b6395c5f5e4fe85414c4d8c"
Nov 03 20:26:19.898199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.898179    5424 scope.go:122] "RemoveContainer" containerID="985bffde5d57536c2029b61c159497866733cba362fdf270c3618f92488d7088"
Nov 03 20:26:19.917100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.917075    5424 scope.go:122] "RemoveContainer" containerID="d8fd864d474b28036103bddde8793d44c6c462b1a6df675916de6cdb5b1d166f"
Nov 03 20:26:19.977569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:19.969190    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13b617e-38de-40e6-9f35-87629d33a97c" path="/var/lib/kubelet/pods/e13b617e-38de-40e6-9f35-87629d33a97c/volumes"
Nov 03 20:26:20.064612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.063602    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"]
Nov 03 20:26:20.076730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.076701    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ae90ca-cnkk6"]
Nov 03 20:26:20.871165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.871131    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.871488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.871193    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 20:26:20.990562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.989257    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:20.991141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:20.991120    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:21.004257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:21.000937    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:26:21.882970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:21.882940    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:21.884300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:21.884270    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:21.885703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:21.885651    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:26:21.949368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:21.949253    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270d6e52-0b79-4b9c-806a-64f2ca779fa0" path="/var/lib/kubelet/pods/270d6e52-0b79-4b9c-806a-64f2ca779fa0/volumes"
Nov 03 20:26:22.664718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.664684    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:26:22.670893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.670851    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.691190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.689096    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:26:22.781272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781237    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblnc\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781272    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781414    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781469    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781577    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781752    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781873    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.781948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.781929    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.782075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.782063    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883179    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883286    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883323    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883499    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883584    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblnc\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883614    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883702    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883728    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883752    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883801    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883809    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.883880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.883866    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.884072 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.884045    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.884607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.884470    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.885962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.885939    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.886401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.886378    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.903915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.903887    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:22.904152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:22.904131    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblnc\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:23.031846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.031800    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:23.354928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.354887    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" event={"ID":"210a4013-7945-4fd5-81e3-6c0168af8b38","Type":"ContainerStarted","Data":"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"}
Nov 03 20:26:23.355079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.354974    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:23.361283 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:23.361257    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode072a74d_12c7_4927_abfc_bd476d09763e.slice/crio-e775a3a347c8e4d900c8889c178b09276713faa166e411f91fa39e52938f204f WatchSource:0}: Error finding container e775a3a347c8e4d900c8889c178b09276713faa166e411f91fa39e52938f204f: Status 404 returned error can't find the container with id e775a3a347c8e4d900c8889c178b09276713faa166e411f91fa39e52938f204f
Nov 03 20:26:23.361431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.361362    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:26:23.364584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.364551    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:23.364682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.364618    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:23.364682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.364649    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:23.397259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:23.397209    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" podStartSLOduration=3.680883951 podStartE2EDuration="8.397194574s" totalImagesPullingTime="4.716310623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:23.393633996 +0000 UTC m=+1325.602649992" watchObservedRunningTime="2026-11-03 20:26:23.397194574 +0000 UTC m=+1325.606210569"
Nov 03 20:26:24.050916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:24.050860    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"]
Nov 03 20:26:24.055713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:24.055684    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f47421c-hprfc"]
Nov 03 20:26:24.366622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:24.366479    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" event={"ID":"e072a74d-12c7-4927-abfc-bd476d09763e","Type":"ContainerStarted","Data":"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"}
Nov 03 20:26:24.366622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:24.366558    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" event={"ID":"e072a74d-12c7-4927-abfc-bd476d09763e","Type":"ContainerStarted","Data":"e775a3a347c8e4d900c8889c178b09276713faa166e411f91fa39e52938f204f"}
Nov 03 20:26:24.415798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:24.415748    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podStartSLOduration=2.415734379 podStartE2EDuration="2.415734379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:24.411183188 +0000 UTC m=+1326.620199186" watchObservedRunningTime="2026-11-03 20:26:24.415734379 +0000 UTC m=+1326.624750375"
Nov 03 20:26:25.043756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.043721    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:25.045972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.045945    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.046109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.046014    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:26:25.048874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.048849    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"]
Nov 03 20:26:25.054017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.053992    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8028c2-f49vj"]
Nov 03 20:26:25.950200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.950161    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c174cb2-8f31-40e9-b54c-c6e3c14daaca" path="/var/lib/kubelet/pods/5c174cb2-8f31-40e9-b54c-c6e3c14daaca/volumes"
Nov 03 20:26:25.951016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:25.950991    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886aecdc-b29b-40de-ab51-e1ec518bdbf5" path="/var/lib/kubelet/pods/886aecdc-b29b-40de-ab51-e1ec518bdbf5/volumes"
Nov 03 20:26:26.041595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:26.036997    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.041595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:26.037073    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:26:27.048924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.048834    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:26:27.048924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.048894    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:26:27.237556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.235334    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:26:27.265989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.265951    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.274269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.273653    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:26:27.303758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.299684    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:26:27.303758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.299751    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:26:27.303758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.299770    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315581    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315633    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lrv\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315714    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315743    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315772    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315860    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315921    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315963    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.315992    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316018    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316122    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316151    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316179    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316258    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316286    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316341    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2v87\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316364    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.316733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.316597    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418251    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418304    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418339    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418437    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418503    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418568    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418599    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418630    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418738    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418765    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418799    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.418892    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.440012    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.441068    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.443639    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.443726    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2v87\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.443756    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.443821    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.443969    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.444011    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lrv\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.444832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.444803    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.447053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.445418    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.447053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.445682    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.447053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.446188    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.447053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.446417    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.447053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.447000    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.447328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.447241    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.449221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.449194    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.449524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.449493    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.451349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.451324    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.451596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.451580    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.452258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.452239    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.460274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.460254    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.463704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.463678    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lrv\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.464995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.464962    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2v87\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.466017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.465994    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:27.611417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.611329    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:27.650794 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:27.650748    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:28.011498 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:28.011469    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5d83c9_8df4_4e40_ade9_4ff9e4f3d3e3.slice/crio-8751fb769f95c41816440358e50323f1ec3e601cc01a2e32c3214d4c4d39c781 WatchSource:0}: Error finding container 8751fb769f95c41816440358e50323f1ec3e601cc01a2e32c3214d4c4d39c781: Status 404 returned error can't find the container with id 8751fb769f95c41816440358e50323f1ec3e601cc01a2e32c3214d4c4d39c781
Nov 03 20:26:28.012309 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:28.012283    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade2339f_3a0c_47d5_80e3_80f4d66bdbe1.slice/crio-7843f4a27aff751505efd56cacbae9c3932828b5b3f86b68421c8a00b218016b WatchSource:0}: Error finding container 7843f4a27aff751505efd56cacbae9c3932828b5b3f86b68421c8a00b218016b: Status 404 returned error can't find the container with id 7843f4a27aff751505efd56cacbae9c3932828b5b3f86b68421c8a00b218016b
Nov 03 20:26:28.014714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.014696    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:26:28.015148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015108    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.015225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015190    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.015289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015224    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.015469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015433    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.015519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015501    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.015582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.015571    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:28.050915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.050639    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:26:28.050915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.050699    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:26:28.053349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.053324    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:26:28.478467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.478376    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" event={"ID":"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3","Type":"ContainerStarted","Data":"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"}
Nov 03 20:26:28.480112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.480050    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" event={"ID":"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3","Type":"ContainerStarted","Data":"8751fb769f95c41816440358e50323f1ec3e601cc01a2e32c3214d4c4d39c781"}
Nov 03 20:26:28.481915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.481887    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" event={"ID":"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1","Type":"ContainerStarted","Data":"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"}
Nov 03 20:26:28.482189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.482167    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" event={"ID":"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1","Type":"ContainerStarted","Data":"7843f4a27aff751505efd56cacbae9c3932828b5b3f86b68421c8a00b218016b"}
Nov 03 20:26:28.547422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.547346    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podStartSLOduration=1.5473223329999999 podStartE2EDuration="1.547322333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:28.534124301 +0000 UTC m=+1330.743140408" watchObservedRunningTime="2026-11-03 20:26:28.547322333 +0000 UTC m=+1330.756338333"
Nov 03 20:26:28.603797 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.603745    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podStartSLOduration=1.603728905 podStartE2EDuration="1.603728905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:28.597377624 +0000 UTC m=+1330.806393622" watchObservedRunningTime="2026-11-03 20:26:28.603728905 +0000 UTC m=+1330.812744901"
Nov 03 20:26:28.611487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.611444    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:28.659548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:28.658673    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:29.051434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.048975    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:26:29.051434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.049039    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:26:29.401609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.400779    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:26:29.614125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.614086    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:29.614380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.614155    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:29.673129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.673041    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:29.673378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:29.673350    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:30.279552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.268986    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:30.292563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.287219    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:30.322197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.322171    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:26:30.612965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.611974    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:30.612965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.612029    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:30.675598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.669906    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:30.675598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:30.669976    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:31.611865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:31.611834    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:31.612251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:31.611884    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:31.652877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:31.652841    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:31.673056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:31.661380    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:32.611735 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:32.611702    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.612038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:32.612009    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:32.653555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:32.653004    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.653555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:32.653082    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:33.085823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.085780    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"]
Nov 03 20:26:33.091428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.091403    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72b63d2-499rg"]
Nov 03 20:26:33.618984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.618949    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.620497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.620465    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:33.652870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.652745    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.652870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.652812    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:33.935243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:33.935160    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3545e5c2-1832-45d3-a9b4-149568f1b2b6" path="/var/lib/kubelet/pods/3545e5c2-1832-45d3-a9b4-149568f1b2b6/volumes"
Nov 03 20:26:34.612351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:34.612316    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.612554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:34.612382    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:26:34.658556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:34.651435    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.658556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:34.651509    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:26:35.057027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.056974    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"]
Nov 03 20:26:35.061111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.061076    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82cad2b0-hjzs2"]
Nov 03 20:26:35.619586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.618841    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:35.619795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.619775    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:35.620415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.620390    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:26:35.657317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.657288    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:35.657690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.657673    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:35.658482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.658459    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:26:35.944095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:35.943997    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c28a45-1fba-4ead-9d40-4c64aed8eef3" path="/var/lib/kubelet/pods/d3c28a45-1fba-4ead-9d40-4c64aed8eef3/volumes"
Nov 03 20:26:38.067467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:38.067392    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"]
Nov 03 20:26:38.074213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:38.074165    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08df0367-k6xfd"]
Nov 03 20:26:39.998556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:39.998160    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a75021-8b55-4072-b01e-f013dd058674" path="/var/lib/kubelet/pods/e9a75021-8b55-4072-b01e-f013dd058674/volumes"
Nov 03 20:26:42.525154 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:26:42.525112    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:26:43.414712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.414675    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"]
Nov 03 20:26:43.488567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.465906    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.575671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.542379    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:26:43.575671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.561800    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:26:43.593626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.593589    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"]
Nov 03 20:26:43.612084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.612050    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.612265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.612240    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.612484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.612466    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tkf\" (UniqueName: \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.713973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.713889    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tkf\" (UniqueName: \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.714169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.714106    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.714169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.714141    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.715146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.715124    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.748983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.748946    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.774813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.774760    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tkf\" (UniqueName: \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\") pod \"maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:43.897273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:43.897231    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:44.258437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:44.258408    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"]
Nov 03 20:26:44.261166 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:26:44.261136    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16d5f47_ec60_4189_8ade_e1dcb6f18f96.slice/crio-acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130 WatchSource:0}: Error finding container acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130: Status 404 returned error can't find the container with id acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130
Nov 03 20:26:45.032812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.032776    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"]
Nov 03 20:26:45.047639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.047603    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb3fff-xzvp2"]
Nov 03 20:26:45.339927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.339886    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt" event={"ID":"f16d5f47-ec60-4189-8ade-e1dcb6f18f96","Type":"ContainerStarted","Data":"1086fc549d13b697714475c4bb0bbd3ed195907c6151b94ea8226cf995af4c4d"}
Nov 03 20:26:45.340199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.340177    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt" event={"ID":"f16d5f47-ec60-4189-8ade-e1dcb6f18f96","Type":"ContainerStarted","Data":"acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130"}
Nov 03 20:26:45.418550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.412835    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt" podStartSLOduration=2.412802196 podStartE2EDuration="2.412802196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:45.405759677 +0000 UTC m=+1347.614775675" watchObservedRunningTime="2026-11-03 20:26:45.412802196 +0000 UTC m=+1347.621818206"
Nov 03 20:26:45.925829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:45.925479    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f3a9a7-4303-4241-afbc-39e3dab336e7" path="/var/lib/kubelet/pods/a4f3a9a7-4303-4241-afbc-39e3dab336e7/volumes"
Nov 03 20:26:46.352613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:46.352547    5424 generic.go:356] "Generic (PLEG): container finished" podID="f16d5f47-ec60-4189-8ade-e1dcb6f18f96" containerID="1086fc549d13b697714475c4bb0bbd3ed195907c6151b94ea8226cf995af4c4d" exitCode=0
Nov 03 20:26:46.353077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:46.352725    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt" event={"ID":"f16d5f47-ec60-4189-8ade-e1dcb6f18f96","Type":"ContainerDied","Data":"1086fc549d13b697714475c4bb0bbd3ed195907c6151b94ea8226cf995af4c4d"}
Nov 03 20:26:46.460799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:46.460756    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:46.460990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:46.460846    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:47.724174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.722794    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:47.851577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.845943    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\") pod \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") "
Nov 03 20:26:47.851577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.846121    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\") pod \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") "
Nov 03 20:26:47.851577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.846270    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\" (UniqueName: \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\") pod \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\" (UID: \"f16d5f47-ec60-4189-8ade-e1dcb6f18f96\") "
Nov 03 20:26:47.853646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.852267    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca" pod "f16d5f47-ec60-4189-8ade-e1dcb6f18f96" (UID: "f16d5f47-ec60-4189-8ade-e1dcb6f18f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:47.853646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.852415    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf" pod "f16d5f47-ec60-4189-8ade-e1dcb6f18f96" (UID: "f16d5f47-ec60-4189-8ade-e1dcb6f18f96"). InnerVolumeSpecName "kube-api-access-x2tkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:47.854335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.854301    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca" pod "f16d5f47-ec60-4189-8ade-e1dcb6f18f96" (UID: "f16d5f47-ec60-4189-8ade-e1dcb6f18f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:47.964816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.962308    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tkf\" (UniqueName: \"kubernetes.io/projected/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-kube-api-access-x2tkf\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:47.964816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.962351    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:47.964816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:47.962367    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16d5f47-ec60-4189-8ade-e1dcb6f18f96-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:48.392064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:48.392020    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt" event={"ID":"f16d5f47-ec60-4189-8ade-e1dcb6f18f96","Type":"ContainerDied","Data":"acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130"}
Nov 03 20:26:48.392241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:48.392088    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd91ef18279811dced154836b7aba9d3ef68f280a096baa34f471e16e770130"
Nov 03 20:26:48.392241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:48.392162    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"
Nov 03 20:26:48.658809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:48.658769    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:48.659029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:48.658845    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:51.430447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.430407    5424 generic.go:356] "Generic (PLEG): container finished" podID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerID="9070150753b7976af16ba5a5ef726f4114def09f91f554f5c9e6ff01124f1606" exitCode=0
Nov 03 20:26:51.430937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.430472    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerDied","Data":"9070150753b7976af16ba5a5ef726f4114def09f91f554f5c9e6ff01124f1606"}
Nov 03 20:26:51.431022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.431006    5424 scope.go:122] "RemoveContainer" containerID="9070150753b7976af16ba5a5ef726f4114def09f91f554f5c9e6ff01124f1606"
Nov 03 20:26:51.432753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.432731    5424 generic.go:356] "Generic (PLEG): container finished" podID="284ab405-63f0-43b3-91b7-c435b321481e" containerID="90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a" exitCode=0
Nov 03 20:26:51.432894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.432810    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerDied","Data":"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"}
Nov 03 20:26:51.433148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.433118    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:51.433221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.433188    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:51.433268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.433221    5424 scope.go:122] "RemoveContainer" containerID="90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"
Nov 03 20:26:51.433329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.433224    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:51.434968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.434938    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:51.435067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.435002    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:51.435067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:51.435034    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:52.479388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:52.479347    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerStarted","Data":"8283e78fa64fc57fcbf9489665d352aac0b4bcc503ed6a5ea5fae9a6ba1cc121"}
Nov 03 20:26:52.481960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:52.481927    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerStarted","Data":"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"}
Nov 03 20:26:52.737035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:52.736997    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:26:52.737393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:52.737372    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:52.737616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:52.737587    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:53.210056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:53.209977    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:26:53.210191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:53.210152    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:53.210239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:53.210219    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:53.747556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:53.741123    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:53.747556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:53.741182    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:54.213068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.212967    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:54.213371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.213317    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:54.315553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.307184    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:26:54.339792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.339754    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:26:54.772689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.772650    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:54.773279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.773240    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:54.997759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:54.997707    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:55.117438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:55.116617    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:55.232271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:55.232227    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:55.232591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:55.232559    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:55.731565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:55.731423    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:55.731565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:55.731482    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:56.098719 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.098684    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"]
Nov 03 20:26:56.106561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.106355    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-648e0dcf-f2x7t"]
Nov 03 20:26:56.216920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.216881    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:56.217245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.217195    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:56.740682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.740643    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:56.740995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:56.740961    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:57.212489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:57.210456    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:57.212489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:57.210519    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:57.746556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:57.745642    5424 patch_prober.go:29] interesting pod/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:26:57.746556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:57.745703    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:26:57.975010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:57.974972    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a38ebf-e42c-4ba5-84f2-951e50c47ff5" path="/var/lib/kubelet/pods/84a38ebf-e42c-4ba5-84f2-951e50c47ff5/volumes"
Nov 03 20:26:58.209332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:58.209249    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:26:58.209332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:58.209300    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 20:26:58.733654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:58.733624    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:26:58.734170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:58.734152    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:26:58.734919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:58.734894    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:26:59.214673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.214599    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:26:59.215095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.215083    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:26:59.215901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.215883    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:26:59.318309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.318232    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" podUID="da135bcb-3008-4a4f-a77b-34194d81446c" containerName="payload-processing" containerID="cri-o://4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb" gracePeriod=25
Nov 03 20:26:59.341512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.341400    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" podUID="c4b20869-0ef7-45bf-a431-70b89761a9f4" containerName="payload-pre-processing" containerID="cri-o://76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d" gracePeriod=25
Nov 03 20:26:59.648589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.644561    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:26:59.686107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.686081    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:26:59.697474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.697445    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:26:59.698374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698349    5424 cpu_manager.go:414] "containerMap: removing container" podUID="da135bcb-3008-4a4f-a77b-34194d81446c" containerName="payload-processing"
Nov 03 20:26:59.698374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698370    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da135bcb-3008-4a4f-a77b-34194d81446c" containerName="payload-processing"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698405    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c4b20869-0ef7-45bf-a431-70b89761a9f4" containerName="payload-pre-processing"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698411    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b20869-0ef7-45bf-a431-70b89761a9f4" containerName="payload-pre-processing"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698419    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f16d5f47-ec60-4189-8ade-e1dcb6f18f96" containerName="revoke-keys"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698425    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16d5f47-ec60-4189-8ade-e1dcb6f18f96" containerName="revoke-keys"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698613    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="da135bcb-3008-4a4f-a77b-34194d81446c" containerName="payload-processing"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698629    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b20869-0ef7-45bf-a431-70b89761a9f4" containerName="payload-pre-processing"
Nov 03 20:26:59.698642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.698637    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16d5f47-ec60-4189-8ade-e1dcb6f18f96" containerName="revoke-keys"
Nov 03 20:26:59.701401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.701357    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.705568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.705547    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\") pod \"c4b20869-0ef7-45bf-a431-70b89761a9f4\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") "
Nov 03 20:26:59.705877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.705859    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\") pod \"da135bcb-3008-4a4f-a77b-34194d81446c\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") "
Nov 03 20:26:59.705989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.705949    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\" (UniqueName: \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\") pod \"da135bcb-3008-4a4f-a77b-34194d81446c\" (UID: \"da135bcb-3008-4a4f-a77b-34194d81446c\") "
Nov 03 20:26:59.706048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.705992    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\" (UniqueName: \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\") pod \"c4b20869-0ef7-45bf-a431-70b89761a9f4\" (UID: \"c4b20869-0ef7-45bf-a431-70b89761a9f4\") "
Nov 03 20:26:59.706444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706419    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.706573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706515    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.706573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706557    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.706743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706674    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.706798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706757    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.706897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.706828    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.707622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.707599    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume" pod "c4b20869-0ef7-45bf-a431-70b89761a9f4" (UID: "c4b20869-0ef7-45bf-a431-70b89761a9f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:59.707999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.707977    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume" pod "da135bcb-3008-4a4f-a77b-34194d81446c" (UID: "da135bcb-3008-4a4f-a77b-34194d81446c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:59.709625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.708256    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.709625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.708415    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.709625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.708806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvll\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.709625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.709081    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da135bcb-3008-4a4f-a77b-34194d81446c-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:59.709625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.709121    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b20869-0ef7-45bf-a431-70b89761a9f4-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:59.710404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.710340    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll" pod "c4b20869-0ef7-45bf-a431-70b89761a9f4" (UID: "c4b20869-0ef7-45bf-a431-70b89761a9f4"). InnerVolumeSpecName "kube-api-access-6s9ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:59.710404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.710381    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr" pod "da135bcb-3008-4a4f-a77b-34194d81446c" (UID: "da135bcb-3008-4a4f-a77b-34194d81446c"). InnerVolumeSpecName "kube-api-access-7d5nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:59.718925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.718834    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4cdbeb-openshift-default-dockercfg-l46fb\""
Nov 03 20:26:59.725078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.725044    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:26:59.810949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.810854    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.810949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.810934    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811024    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811093    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvll\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811286    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811301    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811363    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811399    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811430    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811571    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811629    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811655    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811661    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.811964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811822    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5nr\" (UniqueName: \"kubernetes.io/projected/da135bcb-3008-4a4f-a77b-34194d81446c-kube-api-access-7d5nr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:59.811964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811855    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9ll\" (UniqueName: \"kubernetes.io/projected/c4b20869-0ef7-45bf-a431-70b89761a9f4-kube-api-access-6s9ll\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:26:59.811964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.811873    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.813864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.813846    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.813954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.813937    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.830118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.830090    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.830247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.830224    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvll\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\") pod \"e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") " pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:26:59.833633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.833609    5424 generic.go:356] "Generic (PLEG): container finished" podID="da135bcb-3008-4a4f-a77b-34194d81446c" containerID="4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb" exitCode=0
Nov 03 20:26:59.833766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.833659    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"
Nov 03 20:26:59.833766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.833691    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" event={"ID":"da135bcb-3008-4a4f-a77b-34194d81446c","Type":"ContainerDied","Data":"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"}
Nov 03 20:26:59.833766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.833749    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2" event={"ID":"da135bcb-3008-4a4f-a77b-34194d81446c","Type":"ContainerDied","Data":"cee6acd537db0e5c6594707a14224783eb553ce4d7a8f944040be8e972715234"}
Nov 03 20:26:59.833938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.833778    5424 scope.go:122] "RemoveContainer" containerID="4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"
Nov 03 20:26:59.835470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.835447    5424 generic.go:356] "Generic (PLEG): container finished" podID="c4b20869-0ef7-45bf-a431-70b89761a9f4" containerID="76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d" exitCode=0
Nov 03 20:26:59.835639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.835475    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" event={"ID":"c4b20869-0ef7-45bf-a431-70b89761a9f4","Type":"ContainerDied","Data":"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"}
Nov 03 20:26:59.835639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.835495    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"
Nov 03 20:26:59.835639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.835566    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj" event={"ID":"c4b20869-0ef7-45bf-a431-70b89761a9f4","Type":"ContainerDied","Data":"defaba414a790a7a69b95fa518ef42594217584e4072cf1134fd7e8663fa2d77"}
Nov 03 20:26:59.846613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.846593    5424 scope.go:122] "RemoveContainer" containerID="4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"
Nov 03 20:26:59.846907 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:26:59.846886    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb\": container with ID starting with 4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb not found: ID does not exist" containerID="4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"
Nov 03 20:26:59.846983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.846917    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb"} err="failed to get container status \"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb\": rpc error: code = NotFound desc = could not find container \"4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb\": container with ID starting with 4d38b4e2a5eaedfc5e9e376dadde50f4d08833eb45183c5341fdcba8d995e9cb not found: ID does not exist"
Nov 03 20:26:59.846983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.846934    5424 scope.go:122] "RemoveContainer" containerID="76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"
Nov 03 20:26:59.858414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.858390    5424 scope.go:122] "RemoveContainer" containerID="76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"
Nov 03 20:26:59.860860 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:26:59.860830    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d\": container with ID starting with 76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d not found: ID does not exist" containerID="76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"
Nov 03 20:26:59.860966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.860871    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d"} err="failed to get container status \"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d\": rpc error: code = NotFound desc = could not find container \"76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d\": container with ID starting with 76f4dd534a59523c4f23eb69952397b7bb5453308163e7729ed87aa95e505d0d not found: ID does not exist"
Nov 03 20:26:59.887649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.887617    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:26:59.892164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.892142    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55b09bd8-67bb5949f5-hwtxj"]
Nov 03 20:26:59.924717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.924687    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b20869-0ef7-45bf-a431-70b89761a9f4" path="/var/lib/kubelet/pods/c4b20869-0ef7-45bf-a431-70b89761a9f4/volumes"
Nov 03 20:26:59.925076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.925059    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:26:59.928853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:26:59.928831    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55b09bd8-d98bb86c8-8wdt2"]
Nov 03 20:27:00.026411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.026370    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:27:00.346510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.346424    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:27:00.350412 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:27:00.350382    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce3a833_2953_46fa_a2f5_19bc942e3cf9.slice/crio-eaf842b06ff26bd05340eadc44eec2cd90d6a614f39fcba80c08466506979079 WatchSource:0}: Error finding container eaf842b06ff26bd05340eadc44eec2cd90d6a614f39fcba80c08466506979079: Status 404 returned error can't find the container with id eaf842b06ff26bd05340eadc44eec2cd90d6a614f39fcba80c08466506979079
Nov 03 20:27:00.354035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.353998    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:00.354142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.354074    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:00.354142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.354106    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:00.846088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.846055    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerStarted","Data":"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"}
Nov 03 20:27:00.846613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.846107    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerStarted","Data":"eaf842b06ff26bd05340eadc44eec2cd90d6a614f39fcba80c08466506979079"}
Nov 03 20:27:00.890045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:00.889994    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podStartSLOduration=12.889980673 podStartE2EDuration="12.889980673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:00.887099645 +0000 UTC m=+1363.096115650" watchObservedRunningTime="2026-11-03 20:27:00.889980673 +0000 UTC m=+1363.098996668"
Nov 03 20:27:01.027115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:01.026741    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:27:01.053275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:01.052883    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:27:01.862598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:01.862499    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:27:01.864110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:01.864083    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:27:01.925652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:01.925611    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da135bcb-3008-4a4f-a77b-34194d81446c" path="/var/lib/kubelet/pods/da135bcb-3008-4a4f-a77b-34194d81446c/volumes"
Nov 03 20:27:02.719275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:02.719239    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"]
Nov 03 20:27:02.733442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:02.733411    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55b09bd8-drhdt"]
Nov 03 20:27:03.923240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:03.923205    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16d5f47-ec60-4189-8ade-e1dcb6f18f96" path="/var/lib/kubelet/pods/f16d5f47-ec60-4189-8ade-e1dcb6f18f96/volumes"
Nov 03 20:27:09.939550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:09.939501    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:27:09.939932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:09.939774    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" containerID="cri-o://8283e78fa64fc57fcbf9489665d352aac0b4bcc503ed6a5ea5fae9a6ba1cc121" gracePeriod=30
Nov 03 20:27:11.031515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:11.031477    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"]
Nov 03 20:27:11.043787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:11.043748    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6daa7982-frsdr"]
Nov 03 20:27:11.930100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:11.930067    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676d2f6f-0288-4a1f-ab92-8dd64aa2295c" path="/var/lib/kubelet/pods/676d2f6f-0288-4a1f-ab92-8dd64aa2295c/volumes"
Nov 03 20:27:12.288462 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:27:12.288427    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:27:14.218114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:14.218078    5424 patch_prober.go:29] interesting pod/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:14.218568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:14.218145    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:15.021223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.021195    5424 generic.go:356] "Generic (PLEG): container finished" podID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerID="8283e78fa64fc57fcbf9489665d352aac0b4bcc503ed6a5ea5fae9a6ba1cc121" exitCode=0
Nov 03 20:27:15.021381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.021267    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerDied","Data":"8283e78fa64fc57fcbf9489665d352aac0b4bcc503ed6a5ea5fae9a6ba1cc121"}
Nov 03 20:27:15.021381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.021336    5424 scope.go:122] "RemoveContainer" containerID="9070150753b7976af16ba5a5ef726f4114def09f91f554f5c9e6ff01124f1606"
Nov 03 20:27:15.640443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.640412    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:27:15.714814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.714692    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:27:15.715791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.715768    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.715791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.715788    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.715990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.715809    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.715990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.715817    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.716102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.716008    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.716102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.716019    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" containerName="istio-proxy"
Nov 03 20:27:15.719273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.719253    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.731051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.731025    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:27:15.735189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.734469    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d1bfb8\""
Nov 03 20:27:15.735189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.734498    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d1bfb8-dockercfg-rdx9w\""
Nov 03 20:27:15.800870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.800829    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801039    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801101    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801142    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801263    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801311    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801339    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:15.801449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801434    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801458    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:15.801605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801564    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801726    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:15.801860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801839    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:15.801920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801875    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\") pod \"e5693a09-6d1f-4d3b-82fc-0354d1193214\" (UID: \"e5693a09-6d1f-4d3b-82fc-0354d1193214\") "
Nov 03 20:27:15.801979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.801557    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:15.802284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.802262    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.802774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.802692    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcd5\" (UniqueName: \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.803169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.803145    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.803268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.803175    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.803268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.803191    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5693a09-6d1f-4d3b-82fc-0354d1193214-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.803268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.803203    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.803268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.803217    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.804404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.804341    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:15.804886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.804862    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:15.804886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.804864    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "kube-api-access-tlsz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:15.805144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.805121    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo" pod "e5693a09-6d1f-4d3b-82fc-0354d1193214" (UID: "e5693a09-6d1f-4d3b-82fc-0354d1193214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:15.906110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.905872    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcd5\" (UniqueName: \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.908806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.908759    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.914858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.909038    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.914858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.909068    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.914858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.909082    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlsz6\" (UniqueName: \"kubernetes.io/projected/e5693a09-6d1f-4d3b-82fc-0354d1193214-kube-api-access-tlsz6\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.914858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.909095    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5693a09-6d1f-4d3b-82fc-0354d1193214-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:15.914858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.909973    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:15.925684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:15.925661    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcd5\" (UniqueName: \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\") pod \"payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:16.032228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.032189    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw" event={"ID":"e5693a09-6d1f-4d3b-82fc-0354d1193214","Type":"ContainerDied","Data":"02ba6cb8d3991be2dda6a4462b5461a6eba5ffce72fb38155ee7055ad14f143e"}
Nov 03 20:27:16.032228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.032212    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"
Nov 03 20:27:16.032476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.032248    5424 scope.go:122] "RemoveContainer" containerID="8283e78fa64fc57fcbf9489665d352aac0b4bcc503ed6a5ea5fae9a6ba1cc121"
Nov 03 20:27:16.045985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.045963    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:16.088264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.088235    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:27:16.093892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.093865    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55b09bd8-openshift-default-668bc5494f-g5ztw"]
Nov 03 20:27:16.360503 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:27:16.360450    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e42741_62a4_4a34_a9a2_086f76971297.slice/crio-750610490b46c450bd04bd4f4ebfcc59cf6ed1ad87a6903150278bf2c15e0beb WatchSource:0}: Error finding container 750610490b46c450bd04bd4f4ebfcc59cf6ed1ad87a6903150278bf2c15e0beb: Status 404 returned error can't find the container with id 750610490b46c450bd04bd4f4ebfcc59cf6ed1ad87a6903150278bf2c15e0beb
Nov 03 20:27:16.362189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:16.362165    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:27:17.042746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:17.042709    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" event={"ID":"e7e42741-62a4-4a34-a9a2-086f76971297","Type":"ContainerStarted","Data":"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"}
Nov 03 20:27:17.043200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:17.042769    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:17.043200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:17.042798    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" event={"ID":"e7e42741-62a4-4a34-a9a2-086f76971297","Type":"ContainerStarted","Data":"750610490b46c450bd04bd4f4ebfcc59cf6ed1ad87a6903150278bf2c15e0beb"}
Nov 03 20:27:17.083583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:17.083511    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" podStartSLOduration=9.083495948 podStartE2EDuration="9.083495948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:17.081390691 +0000 UTC m=+1379.290406688" watchObservedRunningTime="2026-11-03 20:27:17.083495948 +0000 UTC m=+1379.292511978"
Nov 03 20:27:17.923961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:17.923927    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5693a09-6d1f-4d3b-82fc-0354d1193214" path="/var/lib/kubelet/pods/e5693a09-6d1f-4d3b-82fc-0354d1193214/volumes"
Nov 03 20:27:19.541003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.540968    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"]
Nov 03 20:27:19.545278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.545258    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.562851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.562823    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"]
Nov 03 20:27:19.569939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.569910    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:27:19.570093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.569970    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:27:19.579601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.579570    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpzj\" (UniqueName: \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.580073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.580015    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.580243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.580117    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.681885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.681768    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.682083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.681968    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.682227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.682200    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpzj\" (UniqueName: \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.685069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.683092    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.688511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.688489    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.711597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.711560    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpzj\" (UniqueName: \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-9klgc\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:19.860375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:19.860287    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:20.166655 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:27:20.166622    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91bfd15_eb3e_45db_a9d3_fa70742e966a.slice/crio-f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585 WatchSource:0}: Error finding container f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585: Status 404 returned error can't find the container with id f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585
Nov 03 20:27:20.168149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.168122    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"]
Nov 03 20:27:20.283445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.283415    5424 scope.go:122] "RemoveContainer" containerID="35ed5969a6cfcd8eb3b5ecff58b98207dcd4904e130e85d1a65a7256d64c9cce"
Nov 03 20:27:20.294573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.294552    5424 scope.go:122] "RemoveContainer" containerID="4dc3746237da5b0efa60d3773965c275c89711f2797e988656da65b4b982f9c5"
Nov 03 20:27:20.306346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.306325    5424 scope.go:122] "RemoveContainer" containerID="26feefa7b1688ecbf42e1cd94869a0dbdec4d8ba21350d0a6ca65a834f1c13bc"
Nov 03 20:27:20.317687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.317671    5424 scope.go:122] "RemoveContainer" containerID="2b76bc5693184387967eca6da846c70d9e336bcaad4245ea38ed240e785590ad"
Nov 03 20:27:20.338194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.338171    5424 scope.go:122] "RemoveContainer" containerID="2c7cde5a36b7287a1732ba28aa869c2f89ecf4791da9269f2df6407b0ab3892e"
Nov 03 20:27:20.348265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.348244    5424 scope.go:122] "RemoveContainer" containerID="11f4f1466535fdc7eb5b37ce0b609f46c1f4ee16bc408658b019d0f31ddd927c"
Nov 03 20:27:20.359817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.359790    5424 scope.go:122] "RemoveContainer" containerID="6e152415c9dc37e62228509f6cde5e2e2cab92bf5593c12a7e5b524f732ef10c"
Nov 03 20:27:20.371213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.371192    5424 scope.go:122] "RemoveContainer" containerID="a62e12b8e41d82bb4ac7d4e518f3c65ff526d4897a3b85a13aaac6ffdc523f90"
Nov 03 20:27:20.382210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:20.382191    5424 scope.go:122] "RemoveContainer" containerID="ef38861cd07b6a7024610bf53fd53d1576eb917c2c61b7cf6dac2a279a0a9061"
Nov 03 20:27:21.088658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:21.088625    5424 generic.go:356] "Generic (PLEG): container finished" podID="b91bfd15-eb3e-45db-a9d3-fa70742e966a" containerID="3a0239ae42585bba6b5758f403931bda7b7c42edc3ebc4957b06dee8c49ec6b0" exitCode=0
Nov 03 20:27:21.089048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:21.088702    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc" event={"ID":"b91bfd15-eb3e-45db-a9d3-fa70742e966a","Type":"ContainerDied","Data":"3a0239ae42585bba6b5758f403931bda7b7c42edc3ebc4957b06dee8c49ec6b0"}
Nov 03 20:27:21.089048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:21.088764    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc" event={"ID":"b91bfd15-eb3e-45db-a9d3-fa70742e966a","Type":"ContainerStarted","Data":"f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585"}
Nov 03 20:27:22.280699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.280669    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:22.338486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.338452    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\" (UniqueName: \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\") pod \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") "
Nov 03 20:27:22.338694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.338601    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\") pod \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") "
Nov 03 20:27:22.338879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.338858    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\") pod \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\" (UID: \"b91bfd15-eb3e-45db-a9d3-fa70742e966a\") "
Nov 03 20:27:22.339242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.339213    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca" pod "b91bfd15-eb3e-45db-a9d3-fa70742e966a" (UID: "b91bfd15-eb3e-45db-a9d3-fa70742e966a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:22.339754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.339734    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:22.340942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.340908    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca" pod "b91bfd15-eb3e-45db-a9d3-fa70742e966a" (UID: "b91bfd15-eb3e-45db-a9d3-fa70742e966a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:22.343393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.342043    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj" pod "b91bfd15-eb3e-45db-a9d3-fa70742e966a" (UID: "b91bfd15-eb3e-45db-a9d3-fa70742e966a"). InnerVolumeSpecName "kube-api-access-vgpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:22.442094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.442060    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgpzj\" (UniqueName: \"kubernetes.io/projected/b91bfd15-eb3e-45db-a9d3-fa70742e966a-kube-api-access-vgpzj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:22.442094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:22.442090    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b91bfd15-eb3e-45db-a9d3-fa70742e966a-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.106993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:23.106964    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"
Nov 03 20:27:23.107199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:23.106999    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc" event={"ID":"b91bfd15-eb3e-45db-a9d3-fa70742e966a","Type":"ContainerDied","Data":"f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585"}
Nov 03 20:27:23.107199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:23.107045    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f02f0e023152532de6a1e2cae890e01eb0b1fb552148ad87b4fe2b288d7aa585"
Nov 03 20:27:28.065160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:28.065130    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:27:30.113049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:30.113019    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:27:30.686997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:30.686961    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:27:30.687237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:30.687212    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" containerID="cri-o://1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7" gracePeriod=30
Nov 03 20:27:34.820284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:34.820225    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:34.822014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:34.821961    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:35.113404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.113314    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" podUID="b7291627-dc61-43d8-a726-4272310af44e" containerName="payload-processing" containerID="cri-o://c91a25952376799fc69aac1125a1578d5dd6dffe208044583822cb68e6cd496b" gracePeriod=25
Nov 03 20:27:35.272614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.272577    5424 generic.go:356] "Generic (PLEG): container finished" podID="b7291627-dc61-43d8-a726-4272310af44e" containerID="c91a25952376799fc69aac1125a1578d5dd6dffe208044583822cb68e6cd496b" exitCode=0
Nov 03 20:27:35.272776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.272624    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" event={"ID":"b7291627-dc61-43d8-a726-4272310af44e","Type":"ContainerDied","Data":"c91a25952376799fc69aac1125a1578d5dd6dffe208044583822cb68e6cd496b"}
Nov 03 20:27:35.412044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.412013    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:27:35.525047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.525014    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\" (UniqueName: \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\") pod \"b7291627-dc61-43d8-a726-4272310af44e\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") "
Nov 03 20:27:35.525218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.525139    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\") pod \"b7291627-dc61-43d8-a726-4272310af44e\" (UID: \"b7291627-dc61-43d8-a726-4272310af44e\") "
Nov 03 20:27:35.525583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.525552    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume" pod "b7291627-dc61-43d8-a726-4272310af44e" (UID: "b7291627-dc61-43d8-a726-4272310af44e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:35.525952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.525933    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7291627-dc61-43d8-a726-4272310af44e-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:35.527567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.527495    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82" pod "b7291627-dc61-43d8-a726-4272310af44e" (UID: "b7291627-dc61-43d8-a726-4272310af44e"). InnerVolumeSpecName "kube-api-access-g6x82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:35.627255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.627166    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6x82\" (UniqueName: \"kubernetes.io/projected/b7291627-dc61-43d8-a726-4272310af44e-kube-api-access-g6x82\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:35.949759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:35.949725    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:27:36.034295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034257    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034325    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034447    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034476    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034552    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034583    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034651    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.034747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034717    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.034954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034841    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.034954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034884    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.034954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.034909    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.035107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.035079    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.035164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.035102    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\") pod \"00eb9ba9-5be7-42f8-a745-63ec158d9942\" (UID: \"00eb9ba9-5be7-42f8-a745-63ec158d9942\") "
Nov 03 20:27:36.036023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.035993    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:36.036147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.036130    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.036212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.036151    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.036212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.036164    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.036212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.036177    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.037710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.037613    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "kube-api-access-sbnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:36.037710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.037676    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:36.037909 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.037881    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.038935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.038911    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo" pod "00eb9ba9-5be7-42f8-a745-63ec158d9942" (UID: "00eb9ba9-5be7-42f8-a745-63ec158d9942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:36.142730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.142699    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.142730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.142727    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.142730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.142735    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00eb9ba9-5be7-42f8-a745-63ec158d9942-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.142965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.142744    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnz7\" (UniqueName: \"kubernetes.io/projected/00eb9ba9-5be7-42f8-a745-63ec158d9942-kube-api-access-sbnz7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.142965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.142758    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00eb9ba9-5be7-42f8-a745-63ec158d9942-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.284320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.284289    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"
Nov 03 20:27:36.284320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.284313    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x" event={"ID":"b7291627-dc61-43d8-a726-4272310af44e","Type":"ContainerDied","Data":"4d9e0744dcf6934a77f2f0132a3d32d31bed24c2325d31c63e9cd136aab8a0bf"}
Nov 03 20:27:36.284616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.284369    5424 scope.go:122] "RemoveContainer" containerID="c91a25952376799fc69aac1125a1578d5dd6dffe208044583822cb68e6cd496b"
Nov 03 20:27:36.286115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.286085    5424 generic.go:356] "Generic (PLEG): container finished" podID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerID="1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7" exitCode=0
Nov 03 20:27:36.286217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.286126    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" event={"ID":"00eb9ba9-5be7-42f8-a745-63ec158d9942","Type":"ContainerDied","Data":"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"}
Nov 03 20:27:36.286217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.286141    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"
Nov 03 20:27:36.286217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.286172    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv" event={"ID":"00eb9ba9-5be7-42f8-a745-63ec158d9942","Type":"ContainerDied","Data":"07c288478da67fafb69634bee3d26e58a902e1a11d2066f42e976ff814253425"}
Nov 03 20:27:36.296899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.296869    5424 scope.go:122] "RemoveContainer" containerID="1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"
Nov 03 20:27:36.310290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.310267    5424 scope.go:122] "RemoveContainer" containerID="1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"
Nov 03 20:27:36.311111 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:27:36.311080    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7\": container with ID starting with 1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7 not found: ID does not exist" containerID="1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"
Nov 03 20:27:36.311272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.311122    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7"} err="failed to get container status \"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7\": rpc error: code = NotFound desc = could not find container \"1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7\": container with ID starting with 1eee33b85ed952391721adb18c21c157b3629d6aa5d66a52e1d79b08c6bd51d7 not found: ID does not exist"
Nov 03 20:27:36.343686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.343654    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:27:36.348009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.347982    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-stcjv"]
Nov 03 20:27:36.379356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.379326    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:27:36.390215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:36.386869    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-lwr5x"]
Nov 03 20:27:37.923017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:37.922980    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" path="/var/lib/kubelet/pods/00eb9ba9-5be7-42f8-a745-63ec158d9942/volumes"
Nov 03 20:27:37.923427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:37.923411    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7291627-dc61-43d8-a726-4272310af44e" path="/var/lib/kubelet/pods/b7291627-dc61-43d8-a726-4272310af44e/volumes"
Nov 03 20:27:42.288494 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:27:42.288454    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:27:57.069224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.069191    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"]
Nov 03 20:27:57.069935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.069919    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b7291627-dc61-43d8-a726-4272310af44e" containerName="payload-processing"
Nov 03 20:27:57.069986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.069936    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7291627-dc61-43d8-a726-4272310af44e" containerName="payload-processing"
Nov 03 20:27:57.070024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.069989    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b91bfd15-eb3e-45db-a9d3-fa70742e966a" containerName="revoke-keys"
Nov 03 20:27:57.070024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.069995    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91bfd15-eb3e-45db-a9d3-fa70742e966a" containerName="revoke-keys"
Nov 03 20:27:57.070024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.070007    5424 cpu_manager.go:414] "containerMap: removing container" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy"
Nov 03 20:27:57.070024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.070012    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy"
Nov 03 20:27:57.070145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.070110    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91bfd15-eb3e-45db-a9d3-fa70742e966a" containerName="revoke-keys"
Nov 03 20:27:57.070145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.070120    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="00eb9ba9-5be7-42f8-a745-63ec158d9942" containerName="istio-proxy"
Nov 03 20:27:57.070145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.070127    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7291627-dc61-43d8-a726-4272310af44e" containerName="payload-processing"
Nov 03 20:27:57.073842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.073823    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.092629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.092600    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"]
Nov 03 20:27:57.098773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.098744    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:27:57.098939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.098778    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:27:57.159579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.159543    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cm5f\" (UniqueName: \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.159744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.159730    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.160004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.159984    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.261646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.261597    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.261866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.261683    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cm5f\" (UniqueName: \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.261923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.261888    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.262382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.262355    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.264256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.264234    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.282311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.282284    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cm5f\" (UniqueName: \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\") pod \"maas-api-revoke-sub-c51b5b66-lnmq9\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.389358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.389278    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:27:57.736307 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:27:57.736275    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12996b4_4c39_4a6d_ad6f_ae84a8e3543e.slice/crio-7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a WatchSource:0}: Error finding container 7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a: Status 404 returned error can't find the container with id 7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a
Nov 03 20:27:57.738246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:57.738224    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"]
Nov 03 20:27:58.490599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:58.490564    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9" event={"ID":"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e","Type":"ContainerStarted","Data":"c6f6063ca87a0d1d877a243201f04434ed2e27099686ef0850b0de12c01c32f8"}
Nov 03 20:27:58.491128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:58.490626    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9" event={"ID":"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e","Type":"ContainerStarted","Data":"7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a"}
Nov 03 20:27:58.530800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:58.530749    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9" podStartSLOduration=1.5307353030000002 podStartE2EDuration="1.530735303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:58.528173771 +0000 UTC m=+1420.737189781" watchObservedRunningTime="2026-11-03 20:27:58.530735303 +0000 UTC m=+1420.739751299"
Nov 03 20:27:59.501079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:59.501044    5424 generic.go:356] "Generic (PLEG): container finished" podID="c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" containerID="c6f6063ca87a0d1d877a243201f04434ed2e27099686ef0850b0de12c01c32f8" exitCode=0
Nov 03 20:27:59.501601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:27:59.501133    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9" event={"ID":"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e","Type":"ContainerDied","Data":"c6f6063ca87a0d1d877a243201f04434ed2e27099686ef0850b0de12c01c32f8"}
Nov 03 20:28:00.900986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.900959    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:28:00.991637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.991605    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\") pod \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") "
Nov 03 20:28:00.991834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.991699    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\") pod \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") "
Nov 03 20:28:00.991946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.991928    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\" (UniqueName: \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\") pod \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\" (UID: \"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e\") "
Nov 03 20:28:00.992011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.991947    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca" pod "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" (UID: "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:00.993025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.992998    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:00.994251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.994220    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f" pod "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" (UID: "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e"). InnerVolumeSpecName "kube-api-access-7cm5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:00.994341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:00.994269    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca" pod "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" (UID: "c12996b4-4c39-4a6d-ad6f-ae84a8e3543e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:01.095019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:01.094983    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:01.095019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:01.095015    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cm5f\" (UniqueName: \"kubernetes.io/projected/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e-kube-api-access-7cm5f\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:01.689721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:01.689679    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9" event={"ID":"c12996b4-4c39-4a6d-ad6f-ae84a8e3543e","Type":"ContainerDied","Data":"7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a"}
Nov 03 20:28:01.690284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:01.690267    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7021c120326d233a7f311cbaaf410ee01bf577579159553579eefad47ecd3e7a"
Nov 03 20:28:01.690430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:01.690195    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"
Nov 03 20:28:05.783267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.783231    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"]
Nov 03 20:28:05.784178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.784149    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" containerName="revoke-keys"
Nov 03 20:28:05.784178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.784172    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" containerName="revoke-keys"
Nov 03 20:28:05.784421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.784326    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" containerName="revoke-keys"
Nov 03 20:28:05.787987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.787969    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.808426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.808394    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"]
Nov 03 20:28:05.816816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.816784    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:28:05.817025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.816944    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:28:05.870335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.870280    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.870568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.870424    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.870651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.870575    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zff7\" (UniqueName: \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.972520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.972480    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zff7\" (UniqueName: \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.972752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.972737    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.972817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.972809    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.973511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.973489    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.975469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.975436    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:05.995454 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:05.995420    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zff7\" (UniqueName: \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\") pod \"maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:06.111367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:06.111274    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:06.464569 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:28:06.464517    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211f4470_4f62_431b_9dfc_3daa4d18c2eb.slice/crio-0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d WatchSource:0}: Error finding container 0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d: Status 404 returned error can't find the container with id 0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d
Nov 03 20:28:06.467223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:06.467198    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"]
Nov 03 20:28:06.773868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:06.773831    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn" event={"ID":"211f4470-4f62-431b-9dfc-3daa4d18c2eb","Type":"ContainerStarted","Data":"0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d"}
Nov 03 20:28:07.784484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:07.784445    5424 generic.go:356] "Generic (PLEG): container finished" podID="211f4470-4f62-431b-9dfc-3daa4d18c2eb" containerID="404bd68d538fbabea7ffc54bc0a4e0e63db4e0e523d2eb8c83f1cdd2a8fa8d7a" exitCode=0
Nov 03 20:28:07.784876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:07.784549    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn" event={"ID":"211f4470-4f62-431b-9dfc-3daa4d18c2eb","Type":"ContainerDied","Data":"404bd68d538fbabea7ffc54bc0a4e0e63db4e0e523d2eb8c83f1cdd2a8fa8d7a"}
Nov 03 20:28:08.970116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:08.970087    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:09.071624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.071523    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\") pod \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") "
Nov 03 20:28:09.071865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.071844    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\") pod \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") "
Nov 03 20:28:09.071992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.071891    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\" (UniqueName: \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\") pod \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\" (UID: \"211f4470-4f62-431b-9dfc-3daa4d18c2eb\") "
Nov 03 20:28:09.072300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.072211    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca" pod "211f4470-4f62-431b-9dfc-3daa4d18c2eb" (UID: "211f4470-4f62-431b-9dfc-3daa4d18c2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:09.073007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.072981    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:09.074350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.074321    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca" pod "211f4470-4f62-431b-9dfc-3daa4d18c2eb" (UID: "211f4470-4f62-431b-9dfc-3daa4d18c2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:09.074463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.074436    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7" pod "211f4470-4f62-431b-9dfc-3daa4d18c2eb" (UID: "211f4470-4f62-431b-9dfc-3daa4d18c2eb"). InnerVolumeSpecName "kube-api-access-4zff7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:09.175207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.175121    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zff7\" (UniqueName: \"kubernetes.io/projected/211f4470-4f62-431b-9dfc-3daa4d18c2eb-kube-api-access-4zff7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:09.175207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.175154    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211f4470-4f62-431b-9dfc-3daa4d18c2eb-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:09.804811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.804773    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"
Nov 03 20:28:09.804811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.804799    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn" event={"ID":"211f4470-4f62-431b-9dfc-3daa4d18c2eb","Type":"ContainerDied","Data":"0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d"}
Nov 03 20:28:09.805051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:09.804855    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0caef31609a002a944ed32d893c20088e71ef07bd8c1c2ad30feacb8681e653d"
Nov 03 20:28:12.291875 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:12.291831    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:28:12.838262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.838229    5424 generic.go:356] "Generic (PLEG): container finished" podID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerID="d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae" exitCode=0
Nov 03 20:28:12.838472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.838276    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerDied","Data":"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"}
Nov 03 20:28:12.838835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.838813    5424 scope.go:122] "RemoveContainer" containerID="d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"
Nov 03 20:28:12.841221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.841183    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:12.841338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.841283    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:12.841403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:12.841350    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:13.913970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:13.908086    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerStarted","Data":"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"}
Nov 03 20:28:14.040556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:14.036911    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:28:14.056558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:14.043288    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:14.056558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:14.043368    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:15.032557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:15.028977    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:15.032557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:15.029042    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:16.033133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:16.032672    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:16.033133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:16.032737    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:16.254095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:16.254055    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:28:16.865091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:16.865053    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:28:17.028426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:17.027970    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:17.028426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:17.028047    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:18.030330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:18.030291    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:18.042793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:18.042741    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:19.049292 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:19.049248    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:28:19.049851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:19.049316    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 20:28:20.031310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.031284    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:28:20.032389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.032369    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:28:20.482060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.482031    5424 scope.go:122] "RemoveContainer" containerID="5fd4bd52660672dc9c846e85e540ce074106a56082c94c779818e49b3d01e106"
Nov 03 20:28:20.494280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.494249    5424 scope.go:122] "RemoveContainer" containerID="ff38ca677922c3cda27488196a3c06c1cb23502c25ab7fad8f6f6e8eca568030"
Nov 03 20:28:20.504912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.504892    5424 scope.go:122] "RemoveContainer" containerID="0cfa21abf151923cade4d99da00c81a9aab040490899bbea8ca91a2e09b88721"
Nov 03 20:28:20.515226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.515208    5424 scope.go:122] "RemoveContainer" containerID="be23ab477f846dd4fcd43dd2c67b5c061ff0f4e8ff3d3a176c3f81375458d862"
Nov 03 20:28:20.525500 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.525477    5424 scope.go:122] "RemoveContainer" containerID="aadf449ac4061670e9759665934bfffb22e741aabf5730bf8f9c1d68f68fa9f5"
Nov 03 20:28:20.535326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.535305    5424 scope.go:122] "RemoveContainer" containerID="bfa87f420c8af8e0c4775518a2c7d28fb7a749e438d346804a0e3b539f11ae4e"
Nov 03 20:28:20.545437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.545417    5424 scope.go:122] "RemoveContainer" containerID="f26dc6620968cbd5e4e531f81e85d6b3d14878b45dda2837f3586859bcddabae"
Nov 03 20:28:20.556386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:20.556367    5424 scope.go:122] "RemoveContainer" containerID="cc0383df9ac433f16d1c8267818639209d0c8a32a7b6bb979a0be60ebb68a6e6"
Nov 03 20:28:21.255276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.255234    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" podUID="3f16513b-6b89-4450-9760-259709a68873" containerName="payload-processing" containerID="cri-o://ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04" gracePeriod=25
Nov 03 20:28:21.624295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.624264    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:28:21.735782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.735749    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\" (UniqueName: \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\") pod \"3f16513b-6b89-4450-9760-259709a68873\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") "
Nov 03 20:28:21.735999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.735877    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\") pod \"3f16513b-6b89-4450-9760-259709a68873\" (UID: \"3f16513b-6b89-4450-9760-259709a68873\") "
Nov 03 20:28:21.736301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.736272    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume" pod "3f16513b-6b89-4450-9760-259709a68873" (UID: "3f16513b-6b89-4450-9760-259709a68873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:21.736948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.736927    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f16513b-6b89-4450-9760-259709a68873-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:21.738518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.738492    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7" pod "3f16513b-6b89-4450-9760-259709a68873" (UID: "3f16513b-6b89-4450-9760-259709a68873"). InnerVolumeSpecName "kube-api-access-4zsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:21.843621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:21.840069    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsb7\" (UniqueName: \"kubernetes.io/projected/3f16513b-6b89-4450-9760-259709a68873-kube-api-access-4zsb7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:22.043384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.043351    5424 generic.go:356] "Generic (PLEG): container finished" podID="3f16513b-6b89-4450-9760-259709a68873" containerID="ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04" exitCode=0
Nov 03 20:28:22.043577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.043426    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"
Nov 03 20:28:22.043577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.043439    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" event={"ID":"3f16513b-6b89-4450-9760-259709a68873","Type":"ContainerDied","Data":"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"}
Nov 03 20:28:22.043577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.043489    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls" event={"ID":"3f16513b-6b89-4450-9760-259709a68873","Type":"ContainerDied","Data":"6d02f9a8736150a10f1fdfad4a06d589a503aaaa3682d3b2b44ee4fa25af7a7e"}
Nov 03 20:28:22.043577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.043519    5424 scope.go:122] "RemoveContainer" containerID="ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"
Nov 03 20:28:22.055044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.055023    5424 scope.go:122] "RemoveContainer" containerID="ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"
Nov 03 20:28:22.055350 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:22.055326    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04\": container with ID starting with ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04 not found: ID does not exist" containerID="ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"
Nov 03 20:28:22.055393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.055361    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04"} err="failed to get container status \"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04\": rpc error: code = NotFound desc = could not find container \"ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04\": container with ID starting with ebc6ecb67ec4f7b8503ec94e3c1d8c80fc47e8100a899cb12b8f86dedd58df04 not found: ID does not exist"
Nov 03 20:28:22.097993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.097911    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:28:22.111147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:22.111106    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dc7b94-868db977b4-6fnls"]
Nov 03 20:28:23.921264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:23.921230    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f16513b-6b89-4450-9760-259709a68873" path="/var/lib/kubelet/pods/3f16513b-6b89-4450-9760-259709a68873/volumes"
Nov 03 20:28:28.228263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.228187    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"]
Nov 03 20:28:28.228974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.228956    5424 cpu_manager.go:414] "containerMap: removing container" podUID="211f4470-4f62-431b-9dfc-3daa4d18c2eb" containerName="revoke-keys"
Nov 03 20:28:28.229016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.228977    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211f4470-4f62-431b-9dfc-3daa4d18c2eb" containerName="revoke-keys"
Nov 03 20:28:28.229016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.228989    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3f16513b-6b89-4450-9760-259709a68873" containerName="payload-processing"
Nov 03 20:28:28.229016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.228994    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f16513b-6b89-4450-9760-259709a68873" containerName="payload-processing"
Nov 03 20:28:28.229144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.229135    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="211f4470-4f62-431b-9dfc-3daa4d18c2eb" containerName="revoke-keys"
Nov 03 20:28:28.229177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.229145    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f16513b-6b89-4450-9760-259709a68873" containerName="payload-processing"
Nov 03 20:28:28.234127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.234106    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.249060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.249035    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"]
Nov 03 20:28:28.261313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.261283    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:28:28.261457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.261391    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:28:28.281504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.281477    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmxh\" (UniqueName: \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.281751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.281734    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.281878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.281855    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.383919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.383888    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.384088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.383958    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.384155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.384117    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmxh\" (UniqueName: \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.384652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.384628    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.386395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.386377    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.403830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.403803    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmxh\" (UniqueName: \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\") pod \"maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.549597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.549569    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:28.856422 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:28:28.856394    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79989da0_fabe_4dd0_9335_4d2b0069e0c6.slice/crio-f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069 WatchSource:0}: Error finding container f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069: Status 404 returned error can't find the container with id f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069
Nov 03 20:28:28.857410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:28.857361    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"]
Nov 03 20:28:29.158471 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:29.158442    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89" event={"ID":"79989da0-fabe-4dd0-9335-4d2b0069e0c6","Type":"ContainerStarted","Data":"4cb830cbb384e4c5ace84f710d16cb783df51cbccfeda01761600ecef1265562"}
Nov 03 20:28:29.158594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:29.158542    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89" event={"ID":"79989da0-fabe-4dd0-9335-4d2b0069e0c6","Type":"ContainerStarted","Data":"f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069"}
Nov 03 20:28:29.199489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:29.199431    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89" podStartSLOduration=1.199419529 podStartE2EDuration="1.199419529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:29.196063192 +0000 UTC m=+1451.405079188" watchObservedRunningTime="2026-11-03 20:28:29.199419529 +0000 UTC m=+1451.408435553"
Nov 03 20:28:30.169340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:30.169257    5424 generic.go:356] "Generic (PLEG): container finished" podID="79989da0-fabe-4dd0-9335-4d2b0069e0c6" containerID="4cb830cbb384e4c5ace84f710d16cb783df51cbccfeda01761600ecef1265562" exitCode=0
Nov 03 20:28:30.169743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:30.169337    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89" event={"ID":"79989da0-fabe-4dd0-9335-4d2b0069e0c6","Type":"ContainerDied","Data":"4cb830cbb384e4c5ace84f710d16cb783df51cbccfeda01761600ecef1265562"}
Nov 03 20:28:31.354475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.354446    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:31.452489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.452307    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\") pod \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") "
Nov 03 20:28:31.452697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.452402    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\" (UniqueName: \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\") pod \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") "
Nov 03 20:28:31.452859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.452837    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\") pod \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\" (UID: \"79989da0-fabe-4dd0-9335-4d2b0069e0c6\") "
Nov 03 20:28:31.455920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.455884    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca" pod "79989da0-fabe-4dd0-9335-4d2b0069e0c6" (UID: "79989da0-fabe-4dd0-9335-4d2b0069e0c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:31.477487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.469072    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca" pod "79989da0-fabe-4dd0-9335-4d2b0069e0c6" (UID: "79989da0-fabe-4dd0-9335-4d2b0069e0c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:31.477487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.475457    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh" pod "79989da0-fabe-4dd0-9335-4d2b0069e0c6" (UID: "79989da0-fabe-4dd0-9335-4d2b0069e0c6"). InnerVolumeSpecName "kube-api-access-7pmxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:31.555391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.555359    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:31.555391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.555391    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79989da0-fabe-4dd0-9335-4d2b0069e0c6-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:31.555645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:31.555402    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmxh\" (UniqueName: \"kubernetes.io/projected/79989da0-fabe-4dd0-9335-4d2b0069e0c6-kube-api-access-7pmxh\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:32.186988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.186903    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"
Nov 03 20:28:32.186988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.186922    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89" event={"ID":"79989da0-fabe-4dd0-9335-4d2b0069e0c6","Type":"ContainerDied","Data":"f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069"}
Nov 03 20:28:32.186988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.186980    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f097e99a09ae793acc31b23c0f427cddb64ecfb0ac1d1f150d8875e29659f069"
Nov 03 20:28:32.758697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.758664    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"]
Nov 03 20:28:32.759419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.759402    5424 cpu_manager.go:414] "containerMap: removing container" podUID="79989da0-fabe-4dd0-9335-4d2b0069e0c6" containerName="revoke-keys"
Nov 03 20:28:32.759419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.759416    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79989da0-fabe-4dd0-9335-4d2b0069e0c6" containerName="revoke-keys"
Nov 03 20:28:32.759611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.759550    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="79989da0-fabe-4dd0-9335-4d2b0069e0c6" containerName="revoke-keys"
Nov 03 20:28:32.764596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.764522    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.780190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.780145    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:28:32.780368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.780347    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:28:32.783804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.783780    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"]
Nov 03 20:28:32.884814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.884779    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.885004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.884849    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmt75\" (UniqueName: \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.885004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.884965    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.905011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.904978    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:28:32.910882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.910839    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:28:32.931815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.931787    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:28:32.988209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.988086    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.988426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.988260    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmt75\" (UniqueName: \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.988426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.988331    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.989487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.989462    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:32.995874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:32.995853    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:33.010319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:33.010258    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmt75\" (UniqueName: \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\") pod \"maas-api-revoke-sub-0b84599c-nl7x4\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:33.087521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:33.087488    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:33.610557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:33.604827    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"]
Nov 03 20:28:34.220810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:34.220724    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4" event={"ID":"58333060-939f-476b-b071-53af2ab33053","Type":"ContainerStarted","Data":"104ca5fae0171f2955093ea8d07cdc04cc773dc9b1c66a53b45fbec5d62b562c"}
Nov 03 20:28:35.507951 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:35.498207    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4" event={"ID":"58333060-939f-476b-b071-53af2ab33053","Type":"ContainerStarted","Data":"cd2aee82ffce753a8f5e6190f89bb807bee6fbff9e5ecd34806ef6afb19dfb55"}
Nov 03 20:28:35.536688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:35.536396    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"]
Nov 03 20:28:35.581609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:35.577687    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-9klgc"]
Nov 03 20:28:35.581813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:35.581744    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4" podStartSLOduration=3.581717073 podStartE2EDuration="3.581717073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:35.566397724 +0000 UTC m=+1457.775413722" watchObservedRunningTime="2026-11-03 20:28:35.581717073 +0000 UTC m=+1457.790733070"
Nov 03 20:28:35.925122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:35.925040    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91bfd15-eb3e-45db-a9d3-fa70742e966a" path="/var/lib/kubelet/pods/b91bfd15-eb3e-45db-a9d3-fa70742e966a/volumes"
Nov 03 20:28:36.510372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:36.510330    5424 generic.go:356] "Generic (PLEG): container finished" podID="58333060-939f-476b-b071-53af2ab33053" containerID="cd2aee82ffce753a8f5e6190f89bb807bee6fbff9e5ecd34806ef6afb19dfb55" exitCode=0
Nov 03 20:28:36.510953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:36.510425    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4" event={"ID":"58333060-939f-476b-b071-53af2ab33053","Type":"ContainerDied","Data":"cd2aee82ffce753a8f5e6190f89bb807bee6fbff9e5ecd34806ef6afb19dfb55"}
Nov 03 20:28:37.698617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.698587    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:37.816023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.815933    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\" (UniqueName: \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\") pod \"58333060-939f-476b-b071-53af2ab33053\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") "
Nov 03 20:28:37.816023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.815976    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\") pod \"58333060-939f-476b-b071-53af2ab33053\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") "
Nov 03 20:28:37.816278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.816086    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\") pod \"58333060-939f-476b-b071-53af2ab33053\" (UID: \"58333060-939f-476b-b071-53af2ab33053\") "
Nov 03 20:28:37.816430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.816406    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca" pod "58333060-939f-476b-b071-53af2ab33053" (UID: "58333060-939f-476b-b071-53af2ab33053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:37.816930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.816908    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58333060-939f-476b-b071-53af2ab33053-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:37.818359 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.818330    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca" pod "58333060-939f-476b-b071-53af2ab33053" (UID: "58333060-939f-476b-b071-53af2ab33053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:37.818464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.818401    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75" pod "58333060-939f-476b-b071-53af2ab33053" (UID: "58333060-939f-476b-b071-53af2ab33053"). InnerVolumeSpecName "kube-api-access-rmt75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:37.918893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.918851    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmt75\" (UniqueName: \"kubernetes.io/projected/58333060-939f-476b-b071-53af2ab33053-kube-api-access-rmt75\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:37.918893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:37.918877    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58333060-939f-476b-b071-53af2ab33053-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:38.528509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:38.528481    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"
Nov 03 20:28:38.528729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:38.528519    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4" event={"ID":"58333060-939f-476b-b071-53af2ab33053","Type":"ContainerDied","Data":"104ca5fae0171f2955093ea8d07cdc04cc773dc9b1c66a53b45fbec5d62b562c"}
Nov 03 20:28:38.528729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:38.528583    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="104ca5fae0171f2955093ea8d07cdc04cc773dc9b1c66a53b45fbec5d62b562c"
Nov 03 20:28:38.715773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:38.715745    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:28:42.288870 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:42.288823    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:28:43.716725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:43.716680    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing" containerID="cri-o://8ade6c51abbfd1d15060ce95070a0b595dcfdeb6268f1374c45c1a8f92d91745" gracePeriod=25
Nov 03 20:28:47.930365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:47.930325    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main" containerID="cri-o://42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753" gracePeriod=45
Nov 03 20:28:47.933656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:47.933628    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main" containerID="cri-o://dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2" gracePeriod=45
Nov 03 20:28:47.956227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:47.956186    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" containerID="cri-o://4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7" gracePeriod=45
Nov 03 20:28:48.345649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.345253    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:28:48.405744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.405703    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:28:48.408855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.408832    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:28:48.512832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512569    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.512832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512739    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.512832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512781    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.512832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512806    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.512832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512829    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\" (UniqueName: \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512870    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512900    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512922    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.512972    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513000    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\" (UniqueName: \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513130    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.513244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513175    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513289    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513334    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513353    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\" (UniqueName: \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\") pod \"759a2b86-568a-4b90-8485-2fb28e12a7f2\" (UID: \"759a2b86-568a-4b90-8485-2fb28e12a7f2\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513392    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513418    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\") pod \"4f345309-2a86-4e03-b805-9089c9023b91\" (UID: \"4f345309-2a86-4e03-b805-9089c9023b91\") "
Nov 03 20:28:48.513696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.513474    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\") pod \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\" (UID: \"db11a3fd-7705-4e94-9345-9e22c8bff5f2\") "
Nov 03 20:28:48.515069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.514740    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.515223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.515153    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.516114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.515708    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.516114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.515735    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.516114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.515916    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.516114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.516069    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.516413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.516285    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.517948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.517917    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.518204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.518179    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.518523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.518495    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:48.519768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.519742    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:48.519868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.519843    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "kube-api-access-n5wwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:48.519868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.519859    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:48.519990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.519952    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.521374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.521347    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.522820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.522793    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm" pod "759a2b86-568a-4b90-8485-2fb28e12a7f2" (UID: "759a2b86-568a-4b90-8485-2fb28e12a7f2"). InnerVolumeSpecName "kube-api-access-r5stm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:48.522947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.522836    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n" pod "db11a3fd-7705-4e94-9345-9e22c8bff5f2" (UID: "db11a3fd-7705-4e94-9345-9e22c8bff5f2"). InnerVolumeSpecName "kube-api-access-d9p4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:48.524684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.524616    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm" pod "4f345309-2a86-4e03-b805-9089c9023b91" (UID: "4f345309-2a86-4e03-b805-9089c9023b91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.616629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616596    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f345309-2a86-4e03-b805-9089c9023b91-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616631    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5stm\" (UniqueName: \"kubernetes.io/projected/759a2b86-568a-4b90-8485-2fb28e12a7f2-kube-api-access-r5stm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616645    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616655    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616662    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616669    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616677    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616685    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759a2b86-568a-4b90-8485-2fb28e12a7f2-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616692    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616701    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5wwg\" (UniqueName: \"kubernetes.io/projected/4f345309-2a86-4e03-b805-9089c9023b91-kube-api-access-n5wwg\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616708    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db11a3fd-7705-4e94-9345-9e22c8bff5f2-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616716    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616722    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616730    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616737    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9p4n\" (UniqueName: \"kubernetes.io/projected/db11a3fd-7705-4e94-9345-9e22c8bff5f2-kube-api-access-d9p4n\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616744    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f345309-2a86-4e03-b805-9089c9023b91-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616752    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759a2b86-568a-4b90-8485-2fb28e12a7f2-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.616823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.616758    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db11a3fd-7705-4e94-9345-9e22c8bff5f2-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.814081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.813981    5424 generic.go:356] "Generic (PLEG): container finished" podID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerID="4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7" exitCode=0
Nov 03 20:28:48.814081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.814039    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"
Nov 03 20:28:48.814081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.814055    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" event={"ID":"759a2b86-568a-4b90-8485-2fb28e12a7f2","Type":"ContainerDied","Data":"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"}
Nov 03 20:28:48.814377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.814127    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" event={"ID":"759a2b86-568a-4b90-8485-2fb28e12a7f2","Type":"ContainerDied","Data":"acc71daaef3850a25aa2479381646516a6033620eba635fe321d898594881a95"}
Nov 03 20:28:48.814377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.814151    5424 scope.go:122] "RemoveContainer" containerID="4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"
Nov 03 20:28:48.816750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.816719    5424 generic.go:356] "Generic (PLEG): container finished" podID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerID="42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753" exitCode=0
Nov 03 20:28:48.816863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.816769    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"
Nov 03 20:28:48.817010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.816991    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" event={"ID":"db11a3fd-7705-4e94-9345-9e22c8bff5f2","Type":"ContainerDied","Data":"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"}
Nov 03 20:28:48.817097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.817023    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz" event={"ID":"db11a3fd-7705-4e94-9345-9e22c8bff5f2","Type":"ContainerDied","Data":"caec19c164d6c5a55a20d75d556566f54271acc0c6f423343d18205a5ad3ba65"}
Nov 03 20:28:48.819419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.819395    5424 generic.go:356] "Generic (PLEG): container finished" podID="4f345309-2a86-4e03-b805-9089c9023b91" containerID="dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2" exitCode=0
Nov 03 20:28:48.819524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.819489    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" event={"ID":"4f345309-2a86-4e03-b805-9089c9023b91","Type":"ContainerDied","Data":"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"}
Nov 03 20:28:48.819524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.819516    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5" event={"ID":"4f345309-2a86-4e03-b805-9089c9023b91","Type":"ContainerDied","Data":"e728d330ffbab7a0dc3264cbbdac394e31eb08ded28a5fa3c18f93920eb5891e"}
Nov 03 20:28:48.819761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.819742    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"
Nov 03 20:28:48.834056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.834038    5424 scope.go:122] "RemoveContainer" containerID="4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"
Nov 03 20:28:48.834466 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:48.834397    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7\": container with ID starting with 4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7 not found: ID does not exist" containerID="4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"
Nov 03 20:28:48.834713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.834462    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7"} err="failed to get container status \"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7\": rpc error: code = NotFound desc = could not find container \"4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7\": container with ID starting with 4662ba50d2f778249e7ada82a9a6eeccc7b59b2a3417c44cf0b6cfeb150ff5c7 not found: ID does not exist"
Nov 03 20:28:48.834713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.834503    5424 scope.go:122] "RemoveContainer" containerID="42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"
Nov 03 20:28:48.912455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.911767    5424 scope.go:122] "RemoveContainer" containerID="42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"
Nov 03 20:28:48.914082 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:48.913919    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753\": container with ID starting with 42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753 not found: ID does not exist" containerID="42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"
Nov 03 20:28:48.914082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.913964    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753"} err="failed to get container status \"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753\": rpc error: code = NotFound desc = could not find container \"42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753\": container with ID starting with 42387691d3356068a6931a32ed87d9901e4e2f3dd6d24aeb4da73e76cc365753 not found: ID does not exist"
Nov 03 20:28:48.914082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.913991    5424 scope.go:122] "RemoveContainer" containerID="dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"
Nov 03 20:28:48.931084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.931043    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:28:48.932943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.932800    5424 scope.go:122] "RemoveContainer" containerID="dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"
Nov 03 20:28:48.933275 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:28:48.933250    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2\": container with ID starting with dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2 not found: ID does not exist" containerID="dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"
Nov 03 20:28:48.933348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.933285    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2"} err="failed to get container status \"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2\": rpc error: code = NotFound desc = could not find container \"dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2\": container with ID starting with dd1d2da511b19787fe2c65540e1471b69e0e05ab42ec67e1995150354744d8f2 not found: ID does not exist"
Nov 03 20:28:48.935982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.935960    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh"]
Nov 03 20:28:48.965387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.965356    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:28:48.970024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.970001    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-w1-6fc748-kserve-67d4f574-pq8n5"]
Nov 03 20:28:48.998574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:48.998500    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:28:49.006630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:49.006605    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/distinct-2-w1-6fc748-kserve-548545f5d9-zjkjz"]
Nov 03 20:28:49.268280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:49.268239    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/simulator-w1-6fc748-kserve-b868b885d-82cvh" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 20:28:49.926509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:49.926476    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f345309-2a86-4e03-b805-9089c9023b91" path="/var/lib/kubelet/pods/4f345309-2a86-4e03-b805-9089c9023b91/volumes"
Nov 03 20:28:49.927105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:49.927088    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" path="/var/lib/kubelet/pods/759a2b86-568a-4b90-8485-2fb28e12a7f2/volumes"
Nov 03 20:28:49.927606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:49.927588    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" path="/var/lib/kubelet/pods/db11a3fd-7705-4e94-9345-9e22c8bff5f2/volumes"
Nov 03 20:28:51.895433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:51.895106    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:28:53.050556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:53.049918    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"]
Nov 03 20:28:53.057862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:53.057817    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e3e294-frqd7"]
Nov 03 20:28:53.972563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:53.954390    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffff2e08-dc84-43a7-9c17-6ac22200a3e8" path="/var/lib/kubelet/pods/ffff2e08-dc84-43a7-9c17-6ac22200a3e8/volumes"
Nov 03 20:28:58.813236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.813179    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"]
Nov 03 20:28:58.814991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.814964    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main"
Nov 03 20:28:58.815152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815139    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main"
Nov 03 20:28:58.815290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815280    5424 cpu_manager.go:414] "containerMap: removing container" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main"
Nov 03 20:28:58.815368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815359    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main"
Nov 03 20:28:58.815463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815453    5424 cpu_manager.go:414] "containerMap: removing container" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main"
Nov 03 20:28:58.815559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815548    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main"
Nov 03 20:28:58.815657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815647    5424 cpu_manager.go:414] "containerMap: removing container" podUID="58333060-939f-476b-b071-53af2ab33053" containerName="revoke-keys"
Nov 03 20:28:58.815729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815720    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58333060-939f-476b-b071-53af2ab33053" containerName="revoke-keys"
Nov 03 20:28:58.815979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.815965    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f345309-2a86-4e03-b805-9089c9023b91" containerName="main"
Nov 03 20:28:58.816082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.816072    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="db11a3fd-7705-4e94-9345-9e22c8bff5f2" containerName="main"
Nov 03 20:28:58.816157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.816149    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="759a2b86-568a-4b90-8485-2fb28e12a7f2" containerName="main"
Nov 03 20:28:58.816234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.816226    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="58333060-939f-476b-b071-53af2ab33053" containerName="revoke-keys"
Nov 03 20:28:58.823863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.823844    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"]
Nov 03 20:28:58.826051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.824068    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:58.831215 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.831196    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"]
Nov 03 20:28:58.831437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.831420    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:58.836556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.836511    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"]
Nov 03 20:28:58.848617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.845015    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:28:58.848617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.845711    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:28:58.913408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.913371    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:58.913706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.913684    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkvj\" (UniqueName: \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:58.914048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.914028    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:58.914343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.914323    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:58.914576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.914557    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgdx\" (UniqueName: \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:58.914936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:58.914914    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.020381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.020345    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.020784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.020759    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.020931 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.020916    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkvj\" (UniqueName: \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.021142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.021125    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.021360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.021343    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.021504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.021491    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgdx\" (UniqueName: \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.030565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.023187    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.030834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.023307    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.040551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.037660    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.055581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.051273    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgdx\" (UniqueName: \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\") pod \"maas-api-revoke-sub-c51b1e1f-hg8pq\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.081642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.071255    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkvj\" (UniqueName: \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.081642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.071888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e254cb6-8nd85\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:28:59.159846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.159807    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:28:59.220623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:28:59.212379    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:29:00.085300 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:29:00.085266    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8638cb_8f6a_4b56_8e3c_0df46816b0d6.slice/crio-0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670 WatchSource:0}: Error finding container 0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670: Status 404 returned error can't find the container with id 0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670
Nov 03 20:29:00.094477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.094445    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"]
Nov 03 20:29:00.128330 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:29:00.128301    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5833dc4_263d_4829_b939_6d83454f6b96.slice/crio-dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2 WatchSource:0}: Error finding container dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2: Status 404 returned error can't find the container with id dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2
Nov 03 20:29:00.134083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.134056    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"]
Nov 03 20:29:00.894051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.893948    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:29:00.894234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.894143    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" containerID="cri-o://b9ca39c2247a2acf78a856135a7fbccdbf04c38cc12159c6d147808ef51bdcd3" gracePeriod=30
Nov 03 20:29:00.989814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.989775    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85" event={"ID":"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6","Type":"ContainerStarted","Data":"365d1dc4f41525c6af8e28f0b43ac404cc3e6b28c348ddd890cf0a8eda39f71d"}
Nov 03 20:29:00.989998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.989844    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85" event={"ID":"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6","Type":"ContainerStarted","Data":"0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670"}
Nov 03 20:29:00.991644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.991593    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq" event={"ID":"d5833dc4-263d-4829-b939-6d83454f6b96","Type":"ContainerStarted","Data":"02f5e76335cf9722e3f009211695ba387302153c98f71c370707aa0a9a61e8fb"}
Nov 03 20:29:00.991770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:00.991657    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq" event={"ID":"d5833dc4-263d-4829-b939-6d83454f6b96","Type":"ContainerStarted","Data":"dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2"}
Nov 03 20:29:01.035091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:01.035041    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85" podStartSLOduration=3.035026898 podStartE2EDuration="3.035026898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.033311178 +0000 UTC m=+1483.242327173" watchObservedRunningTime="2026-11-03 20:29:01.035026898 +0000 UTC m=+1483.244042893"
Nov 03 20:29:01.070698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:01.070650    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq" podStartSLOduration=3.070635178 podStartE2EDuration="3.070635178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.069940329 +0000 UTC m=+1483.278956328" watchObservedRunningTime="2026-11-03 20:29:01.070635178 +0000 UTC m=+1483.279651174"
Nov 03 20:29:01.866028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:01.865991    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:29:02.001393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.001363    5424 generic.go:356] "Generic (PLEG): container finished" podID="7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" containerID="365d1dc4f41525c6af8e28f0b43ac404cc3e6b28c348ddd890cf0a8eda39f71d" exitCode=0
Nov 03 20:29:02.001571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.001446    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85" event={"ID":"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6","Type":"ContainerDied","Data":"365d1dc4f41525c6af8e28f0b43ac404cc3e6b28c348ddd890cf0a8eda39f71d"}
Nov 03 20:29:02.003242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.003213    5424 generic.go:356] "Generic (PLEG): container finished" podID="d5833dc4-263d-4829-b939-6d83454f6b96" containerID="02f5e76335cf9722e3f009211695ba387302153c98f71c370707aa0a9a61e8fb" exitCode=0
Nov 03 20:29:02.003322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.003294    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq" event={"ID":"d5833dc4-263d-4829-b939-6d83454f6b96","Type":"ContainerDied","Data":"02f5e76335cf9722e3f009211695ba387302153c98f71c370707aa0a9a61e8fb"}
Nov 03 20:29:02.971434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.971391    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:29:02.971816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:02.971447    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:29:03.242900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.242873    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:29:03.246399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.246378    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:29:03.354828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.354795    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\" (UniqueName: \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\") pod \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") "
Nov 03 20:29:03.355009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.354981    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\") pod \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") "
Nov 03 20:29:03.355121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.355096    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\") pod \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\" (UID: \"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6\") "
Nov 03 20:29:03.355199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.355174    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\") pod \"d5833dc4-263d-4829-b939-6d83454f6b96\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") "
Nov 03 20:29:03.355286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.355268    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\") pod \"d5833dc4-263d-4829-b939-6d83454f6b96\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") "
Nov 03 20:29:03.355374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.355350    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca" pod "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" (UID: "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:03.355430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.355385    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\" (UniqueName: \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\") pod \"d5833dc4-263d-4829-b939-6d83454f6b96\" (UID: \"d5833dc4-263d-4829-b939-6d83454f6b96\") "
Nov 03 20:29:03.356166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.356141    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:03.356990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.356964    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca" pod "d5833dc4-263d-4829-b939-6d83454f6b96" (UID: "d5833dc4-263d-4829-b939-6d83454f6b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:03.357926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.357904    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca" pod "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" (UID: "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:03.358133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.358107    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj" pod "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" (UID: "7e8638cb-8f6a-4b56-8e3c-0df46816b0d6"). InnerVolumeSpecName "kube-api-access-zgkvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:03.358207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.358163    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca" pod "d5833dc4-263d-4829-b939-6d83454f6b96" (UID: "d5833dc4-263d-4829-b939-6d83454f6b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:03.358490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.358470    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx" pod "d5833dc4-263d-4829-b939-6d83454f6b96" (UID: "d5833dc4-263d-4829-b939-6d83454f6b96"). InnerVolumeSpecName "kube-api-access-6zgdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:03.457404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.457372    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:03.457404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.457400    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:03.457625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.457410    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5833dc4-263d-4829-b939-6d83454f6b96-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:03.457625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.457421    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgdx\" (UniqueName: \"kubernetes.io/projected/d5833dc4-263d-4829-b939-6d83454f6b96-kube-api-access-6zgdx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:03.457625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:03.457430    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkvj\" (UniqueName: \"kubernetes.io/projected/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6-kube-api-access-zgkvj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:04.045849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.045822    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"
Nov 03 20:29:04.046293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.045853    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85" event={"ID":"7e8638cb-8f6a-4b56-8e3c-0df46816b0d6","Type":"ContainerDied","Data":"0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670"}
Nov 03 20:29:04.046293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.045914    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e821e89b062b3746188957f18d6461b2aae9a11d6a2600d7642a63122c1d670"
Nov 03 20:29:04.047811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.047678    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"
Nov 03 20:29:04.047811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.047699    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq" event={"ID":"d5833dc4-263d-4829-b939-6d83454f6b96","Type":"ContainerDied","Data":"dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2"}
Nov 03 20:29:04.047811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.047742    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd026682286c7ca8f56aafa7a26b94ea3ac9edc4ea3c9500747071780765d5d2"
Nov 03 20:29:04.050661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.050635    5424 generic.go:356] "Generic (PLEG): container finished" podID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerID="8ade6c51abbfd1d15060ce95070a0b595dcfdeb6268f1374c45c1a8f92d91745" exitCode=0
Nov 03 20:29:04.050768 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.050705    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" event={"ID":"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8","Type":"ContainerDied","Data":"8ade6c51abbfd1d15060ce95070a0b595dcfdeb6268f1374c45c1a8f92d91745"}
Nov 03 20:29:04.052449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.052378    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:29:04.107866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.107839    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\" (UniqueName: \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\") pod \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") "
Nov 03 20:29:04.108015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.107932    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\") pod \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\" (UID: \"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8\") "
Nov 03 20:29:04.108298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.108276    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume" pod "3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" (UID: "3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:04.108701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.108685    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:04.110176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.110155    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q" pod "3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" (UID: "3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8"). InnerVolumeSpecName "kube-api-access-wp47q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:04.210408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:04.210369    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp47q\" (UniqueName: \"kubernetes.io/projected/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8-kube-api-access-wp47q\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:05.060425 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.060394    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"
Nov 03 20:29:05.060906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.060435    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5" event={"ID":"3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8","Type":"ContainerDied","Data":"dbda2ab3c355e6f7a497ecdaff4de059749c5fcf96dabca7830e77cc64cd23a1"}
Nov 03 20:29:05.060906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.060506    5424 scope.go:122] "RemoveContainer" containerID="8ade6c51abbfd1d15060ce95070a0b595dcfdeb6268f1374c45c1a8f92d91745"
Nov 03 20:29:05.113503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.113473    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:29:05.125565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.125522    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6fc748-cbbb4f4c5-tkmt5"]
Nov 03 20:29:05.922733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:05.922701    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" path="/var/lib/kubelet/pods/3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8/volumes"
Nov 03 20:29:06.074666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.074638    5424 generic.go:356] "Generic (PLEG): container finished" podID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerID="b9ca39c2247a2acf78a856135a7fbccdbf04c38cc12159c6d147808ef51bdcd3" exitCode=0
Nov 03 20:29:06.075207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.074946    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" event={"ID":"4e2b8450-64ff-4315-8bb4-cf421e902e43","Type":"ContainerDied","Data":"b9ca39c2247a2acf78a856135a7fbccdbf04c38cc12159c6d147808ef51bdcd3"}
Nov 03 20:29:06.204894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.204865    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:29:06.248601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.248348    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.253556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.248958    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.254038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.254009    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.254302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.254276    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.261479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.261455    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:06.261935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.261912    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:06.262098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.261959    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.262202    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.262339    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.262377    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.262501    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.262566    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\") pod \"4e2b8450-64ff-4315-8bb4-cf421e902e43\" (UID: \"4e2b8450-64ff-4315-8bb4-cf421e902e43\") "
Nov 03 20:29:06.263803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.263766    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:06.264688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.264648    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:06.273100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.270781    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.273100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.270818    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.273100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.270831    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8450-64ff-4315-8bb4-cf421e902e43-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.273100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.270846    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.273100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.270869    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.310555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.296124    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "kube-api-access-zbvg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:06.310555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.297620    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:06.310555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.299607    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:06.310555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.305028    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy" pod "4e2b8450-64ff-4315-8bb4-cf421e902e43" (UID: "4e2b8450-64ff-4315-8bb4-cf421e902e43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:06.373519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.373482    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.373519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.373521    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.373758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.373557    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e2b8450-64ff-4315-8bb4-cf421e902e43-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:06.373758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:06.373569    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbvg7\" (UniqueName: \"kubernetes.io/projected/4e2b8450-64ff-4315-8bb4-cf421e902e43-kube-api-access-zbvg7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:07.086319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.086286    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk" event={"ID":"4e2b8450-64ff-4315-8bb4-cf421e902e43","Type":"ContainerDied","Data":"7f8612b145fa57031ed8374d13563df91941617a0e3d9720bab0e5c7356cfe30"}
Nov 03 20:29:07.086818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.086342    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"
Nov 03 20:29:07.086818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.086347    5424 scope.go:122] "RemoveContainer" containerID="b9ca39c2247a2acf78a856135a7fbccdbf04c38cc12159c6d147808ef51bdcd3"
Nov 03 20:29:07.171736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.171704    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:29:07.178831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.178652    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9qmfk"]
Nov 03 20:29:07.922302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:07.922270    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" path="/var/lib/kubelet/pods/4e2b8450-64ff-4315-8bb4-cf421e902e43/volumes"
Nov 03 20:29:12.288649 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:12.288612    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:29:20.636758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:20.636719    5424 scope.go:122] "RemoveContainer" containerID="3deb2cbe0d911b4d8b2f5d18b9c7bd0e3c0ca72682e312191d0e9dfd28059ccd"
Nov 03 20:29:20.647434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:20.647413    5424 scope.go:122] "RemoveContainer" containerID="4c9c9e4b944c47b7c11b08432acaf867d9cd793d72fdab1e76f1f3962cde9efc"
Nov 03 20:29:20.657757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:20.657735    5424 scope.go:122] "RemoveContainer" containerID="6833faf51a2cfd9f1c8c1dbf9fb2e0d6f83b5bea24d979bd2c1b97c31d181b20"
Nov 03 20:29:27.296219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:27.296187    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"]
Nov 03 20:29:27.301621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:27.301590    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc7b94-t2vsn"]
Nov 03 20:29:27.924411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:27.924371    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211f4470-4f62-431b-9dfc-3daa4d18c2eb" path="/var/lib/kubelet/pods/211f4470-4f62-431b-9dfc-3daa4d18c2eb/volumes"
Nov 03 20:29:35.079866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.079815    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:29:35.081359 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.081316    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy" containerID="cri-o://186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03" gracePeriod=30
Nov 03 20:29:35.649982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.649950    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"]
Nov 03 20:29:35.651847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.651819    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy"
Nov 03 20:29:35.652037 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652021    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy"
Nov 03 20:29:35.652168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652155    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing"
Nov 03 20:29:35.652261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652249    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing"
Nov 03 20:29:35.652430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652417    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" containerName="revoke-keys"
Nov 03 20:29:35.652521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652509    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" containerName="revoke-keys"
Nov 03 20:29:35.652667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652654    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d5833dc4-263d-4829-b939-6d83454f6b96" containerName="revoke-keys"
Nov 03 20:29:35.652763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.652752    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5833dc4-263d-4829-b939-6d83454f6b96" containerName="revoke-keys"
Nov 03 20:29:35.653060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.653041    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" containerName="revoke-keys"
Nov 03 20:29:35.654399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.654382    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5833dc4-263d-4829-b939-6d83454f6b96" containerName="revoke-keys"
Nov 03 20:29:35.654586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.654568    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b8450-64ff-4315-8bb4-cf421e902e43" containerName="istio-proxy"
Nov 03 20:29:35.654867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.654853    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d65d9ce-28f5-43c4-a7e7-a06ffef1a1b8" containerName="payload-pre-processing"
Nov 03 20:29:35.893169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.893056    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"]
Nov 03 20:29:35.893369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.893182    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:35.940188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.928208    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:29:35.940188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:35.928978    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:29:36.116196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.114674    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz565\" (UniqueName: \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.116196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.114929    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.116196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.115158    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.252559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.241204    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.252559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.241346    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz565\" (UniqueName: \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.252559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.241636    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.252559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.242955    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.289199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.289165    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.310666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.310632    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz565\" (UniqueName: \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\") pod \"maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.366717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.366684    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:36.933757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.921638    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:29:36.933757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:36.921706    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.340606 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:29:37.339168    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50145b6f_5d18_49b6_8cb7_a3b47c0c7356.slice/crio-790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7 WatchSource:0}: Error finding container 790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7: Status 404 returned error can't find the container with id 790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7
Nov 03 20:29:37.371025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:37.370994    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"]
Nov 03 20:29:37.429744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:37.427715    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.977955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:37.977910    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l" event={"ID":"50145b6f-5d18-49b6-8cb7-a3b47c0c7356","Type":"ContainerStarted","Data":"790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7"}
Nov 03 20:29:38.989788 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:38.989749    5424 generic.go:356] "Generic (PLEG): container finished" podID="50145b6f-5d18-49b6-8cb7-a3b47c0c7356" containerID="01cdb5afcb3cf1e33fa9b2ceddd062b6e3eadcfbd487e169df87245fdeeb2e2d" exitCode=0
Nov 03 20:29:38.990193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:38.989837    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l" event={"ID":"50145b6f-5d18-49b6-8cb7-a3b47c0c7356","Type":"ContainerDied","Data":"01cdb5afcb3cf1e33fa9b2ceddd062b6e3eadcfbd487e169df87245fdeeb2e2d"}
Nov 03 20:29:39.725484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:39.725448    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"]
Nov 03 20:29:39.755553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:39.753664    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6fc748-76q89"]
Nov 03 20:29:39.927057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:39.927022    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79989da0-fabe-4dd0-9335-4d2b0069e0c6" path="/var/lib/kubelet/pods/79989da0-fabe-4dd0-9335-4d2b0069e0c6/volumes"
Nov 03 20:29:40.221361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.221315    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:40.273137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.272946    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\" (UniqueName: \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\") pod \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") "
Nov 03 20:29:40.273137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.273028    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\") pod \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") "
Nov 03 20:29:40.273137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.273107    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\") pod \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\" (UID: \"50145b6f-5d18-49b6-8cb7-a3b47c0c7356\") "
Nov 03 20:29:40.274131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.274103    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca" pod "50145b6f-5d18-49b6-8cb7-a3b47c0c7356" (UID: "50145b6f-5d18-49b6-8cb7-a3b47c0c7356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:40.276890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.276855    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565" pod "50145b6f-5d18-49b6-8cb7-a3b47c0c7356" (UID: "50145b6f-5d18-49b6-8cb7-a3b47c0c7356"). InnerVolumeSpecName "kube-api-access-qz565". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:40.280639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.279497    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca" pod "50145b6f-5d18-49b6-8cb7-a3b47c0c7356" (UID: "50145b6f-5d18-49b6-8cb7-a3b47c0c7356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:40.375811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.375777    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz565\" (UniqueName: \"kubernetes.io/projected/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-kube-api-access-qz565\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.375811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.375812    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.376048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.375825    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50145b6f-5d18-49b6-8cb7-a3b47c0c7356-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.397135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.397108    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:29:40.486671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.486619    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.486874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.486777    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.486874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.486819    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.488744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.488709    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.488981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.488953    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.489074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.489039    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.489212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.489180    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.489278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.489257    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.489383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.489369    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\") pod \"284ab405-63f0-43b3-91b7-c435b321481e\" (UID: \"284ab405-63f0-43b3-91b7-c435b321481e\") "
Nov 03 20:29:40.491152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.491115    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:40.491690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.491597    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:40.491690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.491663    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:40.492416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.492394    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:40.492733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.492713    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:40.493484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.493455    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:40.494978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.494691    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:40.495463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.495435    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "kube-api-access-7c6x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:40.498270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.498242    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy" pod "284ab405-63f0-43b3-91b7-c435b321481e" (UID: "284ab405-63f0-43b3-91b7-c435b321481e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592626    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284ab405-63f0-43b3-91b7-c435b321481e-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592656    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592665    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592672    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592681    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592693    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284ab405-63f0-43b3-91b7-c435b321481e-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592706    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.592718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592717    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6x8\" (UniqueName: \"kubernetes.io/projected/284ab405-63f0-43b3-91b7-c435b321481e-kube-api-access-7c6x8\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:40.593110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:40.592726    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284ab405-63f0-43b3-91b7-c435b321481e-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:41.015823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.015788    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"
Nov 03 20:29:41.016019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.015815    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l" event={"ID":"50145b6f-5d18-49b6-8cb7-a3b47c0c7356","Type":"ContainerDied","Data":"790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7"}
Nov 03 20:29:41.016019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.015869    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790ac1205ea361294e8e4033af90de6f67a9f01a8ac48c3fffc5b0912e3f00c7"
Nov 03 20:29:41.018303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.018266    5424 generic.go:356] "Generic (PLEG): container finished" podID="284ab405-63f0-43b3-91b7-c435b321481e" containerID="186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03" exitCode=0
Nov 03 20:29:41.018463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.018338    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"
Nov 03 20:29:41.018463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.018337    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerDied","Data":"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"}
Nov 03 20:29:41.018463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.018418    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr" event={"ID":"284ab405-63f0-43b3-91b7-c435b321481e","Type":"ContainerDied","Data":"cbc6d751adca9f6a6a543921b02fa6aef9ac4f012238ff2d0cd5163e35f5e2b4"}
Nov 03 20:29:41.018463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.018447    5424 scope.go:122] "RemoveContainer" containerID="186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"
Nov 03 20:29:41.038886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.038862    5424 scope.go:122] "RemoveContainer" containerID="90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"
Nov 03 20:29:41.050439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.050423    5424 scope.go:122] "RemoveContainer" containerID="186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"
Nov 03 20:29:41.050884 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:41.050859    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03\": container with ID starting with 186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03 not found: ID does not exist" containerID="186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"
Nov 03 20:29:41.050992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.050892    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03"} err="failed to get container status \"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03\": rpc error: code = NotFound desc = could not find container \"186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03\": container with ID starting with 186879cdb33fc066540d113706a7b4b0c8e05bf846c5434cf5416244a6779d03 not found: ID does not exist"
Nov 03 20:29:41.050992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.050909    5424 scope.go:122] "RemoveContainer" containerID="90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"
Nov 03 20:29:41.051231 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:41.051202    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a\": container with ID starting with 90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a not found: ID does not exist" containerID="90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"
Nov 03 20:29:41.051271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.051237    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a"} err="failed to get container status \"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a\": rpc error: code = NotFound desc = could not find container \"90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a\": container with ID starting with 90d02343825a4d4ea1edca8f958bb9a84aa0ca6388a42de17b0eddc69d826b0a not found: ID does not exist"
Nov 03 20:29:41.081608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.081572    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:29:41.095963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.095932    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dc7b94-openshift-default-5fdf84c7f-bgxbr"]
Nov 03 20:29:41.925670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:41.925624    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284ab405-63f0-43b3-91b7-c435b321481e" path="/var/lib/kubelet/pods/284ab405-63f0-43b3-91b7-c435b321481e/volumes"
Nov 03 20:29:42.287937 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:42.287899    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:29:46.612739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:46.609839    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:29:46.612739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:46.610112    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" containerID="cri-o://9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57" gracePeriod=30
Nov 03 20:29:46.625823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:46.625776    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:29:46.626282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:46.626250    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" containerID="cri-o://ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d" gracePeriod=30
Nov 03 20:29:48.075898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.075862    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:29:48.076314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.075927    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:29:48.106118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.106080    5424 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:29:48.106288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.106161    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:29:48.841087 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.841032    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:29:48.841945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.841922    5424 cpu_manager.go:414] "containerMap: removing container" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:48.841945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.841945    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:48.842100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.842002    5424 cpu_manager.go:414] "containerMap: removing container" podUID="50145b6f-5d18-49b6-8cb7-a3b47c0c7356" containerName="revoke-keys"
Nov 03 20:29:48.842100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.842007    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50145b6f-5d18-49b6-8cb7-a3b47c0c7356" containerName="revoke-keys"
Nov 03 20:29:48.842170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.842107    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:48.842170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.842123    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="50145b6f-5d18-49b6-8cb7-a3b47c0c7356" containerName="revoke-keys"
Nov 03 20:29:48.847804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.847781    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.873162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.868852    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:29:48.920214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920176    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920390    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920424    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920507    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920772 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920745    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920840    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.920928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920905    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.921005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.920988    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.921071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.921024    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2h4\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:48.956278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:48.956240    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c4a71d-openshift-default-dockercfg-gr5wd\""
Nov 03 20:29:49.022361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022323    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022375    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022401    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022485    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022522    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022567    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022702    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022760    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2h4\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.022878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022860    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.023058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022880    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.023058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.022919    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.023167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.023144    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.023347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.023309    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.023347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.023313    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.025094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.025071    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.025331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.025312    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.043298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.043265    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.043453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.043404    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2h4\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.168466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.168362    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:49.531861 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:29:49.531811    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690d50f6_a2ad_433d_b64c_30e409183661.slice/crio-f564202824e08fa3503ab48c3f8f6e9842d8a4210f26c0d8cecc51585e3e5d1d WatchSource:0}: Error finding container f564202824e08fa3503ab48c3f8f6e9842d8a4210f26c0d8cecc51585e3e5d1d: Status 404 returned error can't find the container with id f564202824e08fa3503ab48c3f8f6e9842d8a4210f26c0d8cecc51585e3e5d1d
Nov 03 20:29:49.532111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.532089    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:29:49.543690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.543638    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:49.543902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.543739    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:49.543902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:49.543787    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:50.229287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:50.229224    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" event={"ID":"690d50f6-a2ad-433d-b64c-30e409183661","Type":"ContainerStarted","Data":"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"}
Nov 03 20:29:50.229939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:50.229350    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" event={"ID":"690d50f6-a2ad-433d-b64c-30e409183661","Type":"ContainerStarted","Data":"f564202824e08fa3503ab48c3f8f6e9842d8a4210f26c0d8cecc51585e3e5d1d"}
Nov 03 20:29:50.275409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:50.275320    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podStartSLOduration=2.275299362 podStartE2EDuration="2.275299362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:50.272729045 +0000 UTC m=+1532.481745076" watchObservedRunningTime="2026-11-03 20:29:50.275299362 +0000 UTC m=+1532.484315370"
Nov 03 20:29:51.169556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:51.169466    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:51.179552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:51.175635    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:51.179552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:51.175713    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:52.087706 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.086828    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:29:52.090904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.090877    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:29:52.095198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095170    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095211    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095293    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095415    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095427    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.095716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095636    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095807    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:52.095886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095810    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095874    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.095998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.095973    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.096085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.096068    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.096148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.096124    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\") pod \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\" (UID: \"eb8c72c4-0ee1-4b8e-99d7-caf86d381648\") "
Nov 03 20:29:52.096466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.096425    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.097142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.097115    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.097142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.097136    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.097296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.097145    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.097296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.097159    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.097296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.097172    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.099816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.099780    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:52.100091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.100031    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.100437 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.100413    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "kube-api-access-tn2tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.101688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.101655    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy" pod "eb8c72c4-0ee1-4b8e-99d7-caf86d381648" (UID: "eb8c72c4-0ee1-4b8e-99d7-caf86d381648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.171234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.171198    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:52.171515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.171485    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:52.200183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.200154    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.200515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.200493    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.200779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.200759    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.200927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.200913    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.201092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201062    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.201235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201218    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.201384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201368    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.201689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201670    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.201841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201826    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.201984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201968    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\") pod \"72029332-2bf5-454c-8c57-ea06343d7908\" (UID: \"72029332-2bf5-454c-8c57-ea06343d7908\") "
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.201429    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.208376    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.208826    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.209406    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.209583    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.211120    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:52.216582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.216125    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "kube-api-access-6rq2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.217033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.216895    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token" pod "72029332-2bf5-454c-8c57-ea06343d7908" (UID: "72029332-2bf5-454c-8c57-ea06343d7908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.217287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217233    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2tq\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-kube-api-access-tn2tq\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217259    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217274    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217289    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217302    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217315    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217332    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8c72c4-0ee1-4b8e-99d7-caf86d381648-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217347    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72029332-2bf5-454c-8c57-ea06343d7908-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217359    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72029332-2bf5-454c-8c57-ea06343d7908-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217371    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.217561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.217383    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.273857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.273766    5424 generic.go:356] "Generic (PLEG): container finished" podID="72029332-2bf5-454c-8c57-ea06343d7908" containerID="ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d" exitCode=0
Nov 03 20:29:52.274015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.273857    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"
Nov 03 20:29:52.274015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.273989    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" event={"ID":"72029332-2bf5-454c-8c57-ea06343d7908","Type":"ContainerDied","Data":"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"}
Nov 03 20:29:52.274138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.274042    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx" event={"ID":"72029332-2bf5-454c-8c57-ea06343d7908","Type":"ContainerDied","Data":"f207336e62bf93ae8841a88745bbd42f20e61b26e69542da15df39111991757a"}
Nov 03 20:29:52.274138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.274074    5424 scope.go:122] "RemoveContainer" containerID="ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"
Nov 03 20:29:52.279265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.279231    5424 generic.go:356] "Generic (PLEG): container finished" podID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerID="9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57" exitCode=0
Nov 03 20:29:52.280005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.279983    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"
Nov 03 20:29:52.280195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.280170    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" event={"ID":"eb8c72c4-0ee1-4b8e-99d7-caf86d381648","Type":"ContainerDied","Data":"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"}
Nov 03 20:29:52.280291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.280233    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb" event={"ID":"eb8c72c4-0ee1-4b8e-99d7-caf86d381648","Type":"ContainerDied","Data":"fa8a98e3780a02f7e46d08063f5dff15bed839108aff13014c49c7e379c787f1"}
Nov 03 20:29:52.303014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.297075    5424 scope.go:122] "RemoveContainer" containerID="ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"
Nov 03 20:29:52.303014 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:52.297491    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d\": container with ID starting with ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d not found: ID does not exist" containerID="ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"
Nov 03 20:29:52.303014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.297549    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d"} err="failed to get container status \"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d\": rpc error: code = NotFound desc = could not find container \"ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d\": container with ID starting with ae554b772d8f46ec16892641f631a7000adc93d2f4bce29ac1c8cc951cf4c76d not found: ID does not exist"
Nov 03 20:29:52.303014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.297592    5424 scope.go:122] "RemoveContainer" containerID="9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"
Nov 03 20:29:52.310841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.310818    5424 scope.go:122] "RemoveContainer" containerID="9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"
Nov 03 20:29:52.311320 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:29:52.311300    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57\": container with ID starting with 9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57 not found: ID does not exist" containerID="9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"
Nov 03 20:29:52.311424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.311408    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57"} err="failed to get container status \"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57\": rpc error: code = NotFound desc = could not find container \"9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57\": container with ID starting with 9d7b8363518bbb74b21e1fc38abf5f0c87c7a9c6fb7c20f3de761ee04588ef57 not found: ID does not exist"
Nov 03 20:29:52.350609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.323276    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rq2p\" (UniqueName: \"kubernetes.io/projected/72029332-2bf5-454c-8c57-ea06343d7908-kube-api-access-6rq2p\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.350937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.350899    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.351104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.351089    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72029332-2bf5-454c-8c57-ea06343d7908-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.356795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.356769    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:29:52.361749 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.361699    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-58xpx"]
Nov 03 20:29:52.399280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.399235    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:29:52.405150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:52.405111    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-48bwb"]
Nov 03 20:29:53.169494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:53.169449    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:53.169948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:53.169511    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:53.947772 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:53.947733    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72029332-2bf5-454c-8c57-ea06343d7908" path="/var/lib/kubelet/pods/72029332-2bf5-454c-8c57-ea06343d7908/volumes"
Nov 03 20:29:53.948505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:53.948487    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" path="/var/lib/kubelet/pods/eb8c72c4-0ee1-4b8e-99d7-caf86d381648/volumes"
Nov 03 20:29:54.169162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.169126    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:54.169363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.169199    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:54.452649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.452615    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"]
Nov 03 20:29:54.454295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454235    5424 cpu_manager.go:414] "containerMap: removing container" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy"
Nov 03 20:29:54.454629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454604    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy"
Nov 03 20:29:54.454855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454832    5424 cpu_manager.go:414] "containerMap: removing container" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:54.454961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454866    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:54.454961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454894    5424 cpu_manager.go:414] "containerMap: removing container" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy"
Nov 03 20:29:54.454961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.454902    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy"
Nov 03 20:29:54.455872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.455852    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8c72c4-0ee1-4b8e-99d7-caf86d381648" containerName="istio-proxy"
Nov 03 20:29:54.456043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.456030    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="284ab405-63f0-43b3-91b7-c435b321481e" containerName="istio-proxy"
Nov 03 20:29:54.456163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.456152    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="72029332-2bf5-454c-8c57-ea06343d7908" containerName="istio-proxy"
Nov 03 20:29:54.462780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.462761    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"]
Nov 03 20:29:54.463078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.463012    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.482456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.482426    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:29:54.482626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.482492    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:29:54.599246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.599199    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.599589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.599565    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.599698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.599627    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lnb\" (UniqueName: \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.701906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.701867    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.702154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.702138    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.702200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.702185    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lnb\" (UniqueName: \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.702728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.702672    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.704803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.704778    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.741767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.741729    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lnb\" (UniqueName: \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\") pod \"maas-api-revoke-keys-e2e-mt-eb87f429-bbxht\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:54.783929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:54.783894    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:55.159965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:55.154322    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"]
Nov 03 20:29:55.176481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:55.176423    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:55.176684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:55.176504    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:55.308342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:55.308301    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht" event={"ID":"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce","Type":"ContainerStarted","Data":"e79e95f6c7689980cab8c19cb7e2042229d79c4fb0f2dd6e3a6f43a5d4144c2f"}
Nov 03 20:29:56.202660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:56.193907    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 20:29:56.202660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:56.193982    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 20:29:56.331285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:56.330982    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht" event={"ID":"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce","Type":"ContainerStarted","Data":"45b62d6240a3f8a64dab5baa4953ff82bdd7ac35b084cfd6250150868059005d"}
Nov 03 20:29:56.379923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:56.379867    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht" podStartSLOduration=2.3798515670000002 podStartE2EDuration="2.379851567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:56.375816458 +0000 UTC m=+1538.584832454" watchObservedRunningTime="2026-11-03 20:29:56.379851567 +0000 UTC m=+1538.588867543"
Nov 03 20:29:57.174963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:57.174886    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:57.175419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:57.175402    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:57.176094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:57.176071    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:29:57.343581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:57.343509    5424 generic.go:356] "Generic (PLEG): container finished" podID="ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" containerID="45b62d6240a3f8a64dab5baa4953ff82bdd7ac35b084cfd6250150868059005d" exitCode=0
Nov 03 20:29:57.343581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:57.343572    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht" event={"ID":"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce","Type":"ContainerDied","Data":"45b62d6240a3f8a64dab5baa4953ff82bdd7ac35b084cfd6250150868059005d"}
Nov 03 20:29:58.543114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.543087    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:58.577355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.573950    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\") pod \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") "
Nov 03 20:29:58.577355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.574278    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\") pod \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") "
Nov 03 20:29:58.577355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.574442    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\" (UniqueName: \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\") pod \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\" (UID: \"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce\") "
Nov 03 20:29:58.577355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.576975    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca" pod "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" (UID: "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:58.582167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.582115    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca" pod "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" (UID: "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:58.582310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.582230    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb" pod "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" (UID: "ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce"). InnerVolumeSpecName "kube-api-access-67lnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:58.677961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.677868    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:58.677961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.677907    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:58.677961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:58.677916    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lnb\" (UniqueName: \"kubernetes.io/projected/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce-kube-api-access-67lnb\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:29:59.369295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:59.369267    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"
Nov 03 20:29:59.369563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:59.369299    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht" event={"ID":"ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce","Type":"ContainerDied","Data":"e79e95f6c7689980cab8c19cb7e2042229d79c4fb0f2dd6e3a6f43a5d4144c2f"}
Nov 03 20:29:59.369791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:29:59.369766    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79e95f6c7689980cab8c19cb7e2042229d79c4fb0f2dd6e3a6f43a5d4144c2f"
Nov 03 20:30:00.149439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.149407    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:00.150210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.150193    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" containerName="revoke-keys"
Nov 03 20:30:00.150260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.150212    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" containerName="revoke-keys"
Nov 03 20:30:00.150326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.150316    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" containerName="revoke-keys"
Nov 03 20:30:00.154360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.154324    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.154702 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.154673    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:00.160915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.160886    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:00.161097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.161027    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.165742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.165708    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:00.165863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.165804    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.171178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.171141    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:00.171304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.171257    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.177162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.175652    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.178779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.178752    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:30:00.185658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.185631    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:00.194090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.194042    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:00.201623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.201579    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:00.210658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.210633    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:00.238054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.238018    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:00.251717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.251682    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:00.260122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.257189    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:00.260122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.257352    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.262696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.262669    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.270209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.270184    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:00.275729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.275700    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.278351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.278104    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:00.286062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.286026    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:00.286333 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.286314    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.294009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.293876    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g894x\" (UniqueName: \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.294140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294091    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc76d\" (UniqueName: \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.294210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294155    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42g5h\" (UniqueName: \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.294257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294209    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcsm6\" (UniqueName: \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.294294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294254    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.294330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294299    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltfv\" (UniqueName: \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.294399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294378    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.294481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294463    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.294561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294546    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.294663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294649    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.294972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.294950    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:00.303629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.303602    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:00.315076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.310638    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:00.315076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.312057    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.316972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.316938    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.327295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.327272    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:00.333515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.333494    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:00.338267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.338243    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:00.344210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.344189    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:00.344318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.344296    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.349653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.349631    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:00.355374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.355348    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:00.396030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.395999    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.396231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396079    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.396231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396110    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.396231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396141    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcs2l\" (UniqueName: \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.396362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396293    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g894x\" (UniqueName: \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.396445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396420    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.396623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396481    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmv4x\" (UniqueName: \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.396705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396642    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc76d\" (UniqueName: \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.396705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396670    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvdc\" (UniqueName: \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.396839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396723    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2l5\" (UniqueName: \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.396839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396742    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42g5h\" (UniqueName: \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.396839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396824    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.396994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396833    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.396994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396850    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcsm6\" (UniqueName: \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.396994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396886    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.396994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396924    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltfv\" (UniqueName: \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.396994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.396982    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.397324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397018    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.397324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397087    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.397324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397147    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.397668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397645    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.397668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397658    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.397791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.397743    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.422271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.422192    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42g5h\" (UniqueName: \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.422458 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.422419    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltfv\" (UniqueName: \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\") pod \"maas-api-key-cleanup-29895630-cmkl9\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.422630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.422604    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcsm6\" (UniqueName: \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\") pod \"maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.423159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.423135    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc76d\" (UniqueName: \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\") pod \"maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.423740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.423719    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g894x\" (UniqueName: \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\") pod \"maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.472765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.472722    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:00.498865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.498836    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdgx\" (UniqueName: \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.498998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.498891    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prg59\" (UniqueName: \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.499068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.498996    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.499135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499089    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.499135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499121    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.499248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499230    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.499382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499363    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcs2l\" (UniqueName: \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.499501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499479    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.499653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499508    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.499653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499599    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmv4x\" (UniqueName: \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.499792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499767    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.499847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499810    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvdc\" (UniqueName: \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.499899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499862    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2l5\" (UniqueName: \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.499899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499886    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knzk\" (UniqueName: \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.499991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.499974    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.500684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.500658    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.500919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.500888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.501124 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.500984    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:00.501513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.501491    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.510539 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.510502    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:00.521845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.521548    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmv4x\" (UniqueName: \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\") pod \"maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.525733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.525696    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvdc\" (UniqueName: \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.525964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.525938    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2l5\" (UniqueName: \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.526298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.526250    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcs2l\" (UniqueName: \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.534939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.534877    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:00.559597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.551947    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:00.583298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.582096    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:00.601880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.601845    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdgx\" (UniqueName: \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.602028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.601917    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prg59\" (UniqueName: \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.602028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.602006    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.602243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.602221    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.602338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.602320    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.602410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.602394    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knzk\" (UniqueName: \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.605220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.604129    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.605220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.604212    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.605220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.604926    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.605220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.605000    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:00.640614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.637820    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:00.640614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.639693    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:00.646338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.646269    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prg59\" (UniqueName: \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\") pod \"maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.697986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.697811    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdgx\" (UniqueName: \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\") pod \"maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.721913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.713106    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knzk\" (UniqueName: \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:00.944284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.944197    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:00.951997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.951963    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:00.964595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:00.963167    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:01.047192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.047165    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:01.167154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.166788    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:01.181394 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:01.181351    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ad568_08e6_4811_90e3_719b385d8949.slice/crio-8370ed192e513b00297e2daf2c33bb37dfc5720c23f4a778619d3468d33423be WatchSource:0}: Error finding container 8370ed192e513b00297e2daf2c33bb37dfc5720c23f4a778619d3468d33423be: Status 404 returned error can't find the container with id 8370ed192e513b00297e2daf2c33bb37dfc5720c23f4a778619d3468d33423be
Nov 03 20:30:01.186554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.186506    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:01.205753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.205717    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:01.215688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.215660    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:01.216226 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:01.215904    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617670b7_3a01_49cd_887d_4b1c6da28416.slice/crio-e507a8a9533dd044c03ae4d680cf48b11a89371ed7b881c1b2e46da624ef1f0a WatchSource:0}: Error finding container e507a8a9533dd044c03ae4d680cf48b11a89371ed7b881c1b2e46da624ef1f0a: Status 404 returned error can't find the container with id e507a8a9533dd044c03ae4d680cf48b11a89371ed7b881c1b2e46da624ef1f0a
Nov 03 20:30:01.225863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.225831    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:01.233497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.233216    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:01.256457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.241497    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:01.256457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.254668    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:01.376414 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:01.376384    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aed298_5fe8_4767_ab2a_6cb3c084150e.slice/crio-bbddb691ec881032416cacde5e239d8807a674e078dba90e68b0aacbe4e8f544 WatchSource:0}: Error finding container bbddb691ec881032416cacde5e239d8807a674e078dba90e68b0aacbe4e8f544: Status 404 returned error can't find the container with id bbddb691ec881032416cacde5e239d8807a674e078dba90e68b0aacbe4e8f544
Nov 03 20:30:01.379823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.379798    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:01.418430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.411842    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:01.454394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.454357    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerStarted","Data":"b882cd8e044283aa3905e1b8b37a268bfbca34e63f502b990b9920f89bcdd405"}
Nov 03 20:30:01.456646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.456619    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerStarted","Data":"9859c9db3a16109ca2ee1a3114bbf079ab0ec61be94383267657b53e47d8ba1e"}
Nov 03 20:30:01.458767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.458745    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerStarted","Data":"146b56a3874cb5ed012157fc77713462c090af45eae6096aaab15ecaa0c66de8"}
Nov 03 20:30:01.460948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.460921    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerStarted","Data":"f30d65ad2272b30b60ac79fd56f86cdb4730605a7b2e298212e5c44cded79473"}
Nov 03 20:30:01.462318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.462295    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerStarted","Data":"8370ed192e513b00297e2daf2c33bb37dfc5720c23f4a778619d3468d33423be"}
Nov 03 20:30:01.463490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.463466    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerStarted","Data":"1dbca3b65bb6481558c0fa0d8a7901dade98a2dcb6a569bc3de9112352b334f2"}
Nov 03 20:30:01.465069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.465050    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerStarted","Data":"69856c4a47806a60a13498d57e397bfaa2673bdaad88d2ece1e9ebf6d81e7ab1"}
Nov 03 20:30:01.466447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.466424    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerStarted","Data":"8bb91b531903016a2e123a87e9ea6fa78dc9013e6ecbcd7ab11fe5c8be08b80c"}
Nov 03 20:30:01.467659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.467634    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerStarted","Data":"bbddb691ec881032416cacde5e239d8807a674e078dba90e68b0aacbe4e8f544"}
Nov 03 20:30:01.469156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.469136    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerStarted","Data":"e507a8a9533dd044c03ae4d680cf48b11a89371ed7b881c1b2e46da624ef1f0a"}
Nov 03 20:30:01.501041 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:01.501005    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:01.624492 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:01.624454    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2deeb6c_06b5_4632_a2b8_4b4b3202d9c4.slice/crio-c8deee625e73627fb4b291b23c7067a1816abb99ac2d993af83e0545afaf1b09 WatchSource:0}: Error finding container c8deee625e73627fb4b291b23c7067a1816abb99ac2d993af83e0545afaf1b09: Status 404 returned error can't find the container with id c8deee625e73627fb4b291b23c7067a1816abb99ac2d993af83e0545afaf1b09
Nov 03 20:30:01.625343 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:01.625309    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547a2d0a_888e_48ce_8cb4_a7535fc422b4.slice/crio-cd3249b6d03acc94c54eb0c59531844508b47b9fb45b6fe1b51074e75dcc0128 WatchSource:0}: Error finding container cd3249b6d03acc94c54eb0c59531844508b47b9fb45b6fe1b51074e75dcc0128: Status 404 returned error can't find the container with id cd3249b6d03acc94c54eb0c59531844508b47b9fb45b6fe1b51074e75dcc0128
Nov 03 20:30:02.039659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.039349    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"]
Nov 03 20:30:02.045916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.045884    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eae31f9-dd4p9"]
Nov 03 20:30:02.481248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.481207    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerStarted","Data":"9186419f1417db947fadb18a82f0f139d1d7e4f66f388670d1c1e00f3cff4a3c"}
Nov 03 20:30:02.481822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.481271    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerStarted","Data":"cd3249b6d03acc94c54eb0c59531844508b47b9fb45b6fe1b51074e75dcc0128"}
Nov 03 20:30:02.483453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.483423    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerStarted","Data":"053b5e622d5fcef097bebc65ec1796cda1ffbfcdba4445c97294a3d409bffa0d"}
Nov 03 20:30:02.485770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.485733    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerStarted","Data":"2825a96849dc9e62637cfa20f7d97ff2ed1894e746f0217f28926b1a7d91dcb2"}
Nov 03 20:30:02.487974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.487951    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerStarted","Data":"550bc4cb6d469836ed73cb57d86221322ca569b95b53bb3170750fbbdebf0afa"}
Nov 03 20:30:02.489678 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.489654    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerStarted","Data":"77c0e1c500e9f3b61f8d2045ca83b2b983dcdbec2ab02f82c9535f6d5d11ef9c"}
Nov 03 20:30:02.491355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.491325    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerStarted","Data":"a97e155be0f5cc1370f076bdcba7e76931b05c8c9269f1ec7c013ab3f2e9fa71"}
Nov 03 20:30:02.493729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.493696    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerStarted","Data":"473a51bfb94a29294c56ee92e604ac3de476a6b06afd6dbc23214c5e5595f430"}
Nov 03 20:30:02.493829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.493747    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerStarted","Data":"c8deee625e73627fb4b291b23c7067a1816abb99ac2d993af83e0545afaf1b09"}
Nov 03 20:30:02.495679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.495654    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerStarted","Data":"4a09bf397a895b2d43dc9b6354440246687a24cca8a0b7382cd702466bd1145e"}
Nov 03 20:30:02.497924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.497896    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerStarted","Data":"0121d4eb7d2bf9231d821458139976dfd137dc5c435da94e4008c4dbc8ee020a"}
Nov 03 20:30:02.500077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.500050    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerStarted","Data":"512cbe6534356be9679dba2099af413907bc7466096dbfdcfc1bdf785434b4d3"}
Nov 03 20:30:02.501739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.501710    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerStarted","Data":"bf8a25b31898541b6fca63de43f9b91fc3538ea90ba1de4884436c014f2c393d"}
Nov 03 20:30:02.503308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.503287    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerStarted","Data":"cb9281c68c79759f7c57a483ee90366fa0855a85fc7d752d386bc188f4a9667e"}
Nov 03 20:30:02.567745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.567687    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" podStartSLOduration=2.567670364 podStartE2EDuration="2.567670364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.522113893 +0000 UTC m=+1544.731129890" watchObservedRunningTime="2026-11-03 20:30:02.567670364 +0000 UTC m=+1544.776686402"
Nov 03 20:30:02.569835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.569788    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" podStartSLOduration=2.569772263 podStartE2EDuration="2.569772263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.566002114 +0000 UTC m=+1544.775018138" watchObservedRunningTime="2026-11-03 20:30:02.569772263 +0000 UTC m=+1544.778788262"
Nov 03 20:30:02.609510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.609396    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" podStartSLOduration=2.609375573 podStartE2EDuration="2.609375573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.604401019 +0000 UTC m=+1544.813417017" watchObservedRunningTime="2026-11-03 20:30:02.609375573 +0000 UTC m=+1544.818391571"
Nov 03 20:30:02.651975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.651919    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" podStartSLOduration=2.6519004539999997 podStartE2EDuration="2.651900454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.651464389 +0000 UTC m=+1544.860480389" watchObservedRunningTime="2026-11-03 20:30:02.651900454 +0000 UTC m=+1544.860916451"
Nov 03 20:30:02.696872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.696806    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" podStartSLOduration=2.6967838200000003 podStartE2EDuration="2.69678382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.691834533 +0000 UTC m=+1544.900850529" watchObservedRunningTime="2026-11-03 20:30:02.69678382 +0000 UTC m=+1544.905799819"
Nov 03 20:30:02.734203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.734151    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" podStartSLOduration=2.734135364 podStartE2EDuration="2.734135364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.729562348 +0000 UTC m=+1544.938578349" watchObservedRunningTime="2026-11-03 20:30:02.734135364 +0000 UTC m=+1544.943151361"
Nov 03 20:30:02.805495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.805424    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" podStartSLOduration=2.805401475 podStartE2EDuration="2.805401475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.769872964 +0000 UTC m=+1544.978888990" watchObservedRunningTime="2026-11-03 20:30:02.805401475 +0000 UTC m=+1545.014417489"
Nov 03 20:30:02.846621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.844008    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" podStartSLOduration=2.843987055 podStartE2EDuration="2.843987055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.835016065 +0000 UTC m=+1545.044032245" watchObservedRunningTime="2026-11-03 20:30:02.843987055 +0000 UTC m=+1545.053003049"
Nov 03 20:30:02.877716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.877606    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" podStartSLOduration=2.877588068 podStartE2EDuration="2.877588068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.877117101 +0000 UTC m=+1545.086133099" watchObservedRunningTime="2026-11-03 20:30:02.877588068 +0000 UTC m=+1545.086604067"
Nov 03 20:30:02.920466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.920415    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" podStartSLOduration=2.920398457 podStartE2EDuration="2.920398457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.917098399 +0000 UTC m=+1545.126114394" watchObservedRunningTime="2026-11-03 20:30:02.920398457 +0000 UTC m=+1545.129414453"
Nov 03 20:30:02.959384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:02.959327    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" podStartSLOduration=2.959312616 podStartE2EDuration="2.959312616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.95532765 +0000 UTC m=+1545.164343648" watchObservedRunningTime="2026-11-03 20:30:02.959312616 +0000 UTC m=+1545.168328606"
Nov 03 20:30:03.004468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.004410    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" podStartSLOduration=3.004393809 podStartE2EDuration="3.004393809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.998568308 +0000 UTC m=+1545.207584304" watchObservedRunningTime="2026-11-03 20:30:03.004393809 +0000 UTC m=+1545.213409806"
Nov 03 20:30:03.514834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.514800    5424 generic.go:356] "Generic (PLEG): container finished" podID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerID="512cbe6534356be9679dba2099af413907bc7466096dbfdcfc1bdf785434b4d3" exitCode=7
Nov 03 20:30:03.515342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.514919    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerDied","Data":"512cbe6534356be9679dba2099af413907bc7466096dbfdcfc1bdf785434b4d3"}
Nov 03 20:30:03.515427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.515322    5424 scope.go:122] "RemoveContainer" containerID="512cbe6534356be9679dba2099af413907bc7466096dbfdcfc1bdf785434b4d3"
Nov 03 20:30:03.516802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.516773    5424 generic.go:356] "Generic (PLEG): container finished" podID="617670b7-3a01-49cd-887d-4b1c6da28416" containerID="bf8a25b31898541b6fca63de43f9b91fc3538ea90ba1de4884436c014f2c393d" exitCode=60
Nov 03 20:30:03.516916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.516860    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerDied","Data":"bf8a25b31898541b6fca63de43f9b91fc3538ea90ba1de4884436c014f2c393d"}
Nov 03 20:30:03.517135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.517122    5424 scope.go:122] "RemoveContainer" containerID="bf8a25b31898541b6fca63de43f9b91fc3538ea90ba1de4884436c014f2c393d"
Nov 03 20:30:03.519575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.519344    5424 generic.go:356] "Generic (PLEG): container finished" podID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerID="cb9281c68c79759f7c57a483ee90366fa0855a85fc7d752d386bc188f4a9667e" exitCode=60
Nov 03 20:30:03.519575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.519388    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerDied","Data":"cb9281c68c79759f7c57a483ee90366fa0855a85fc7d752d386bc188f4a9667e"}
Nov 03 20:30:03.519731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.519642    5424 scope.go:122] "RemoveContainer" containerID="cb9281c68c79759f7c57a483ee90366fa0855a85fc7d752d386bc188f4a9667e"
Nov 03 20:30:03.522274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.522244    5424 generic.go:356] "Generic (PLEG): container finished" podID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerID="9186419f1417db947fadb18a82f0f139d1d7e4f66f388670d1c1e00f3cff4a3c" exitCode=60
Nov 03 20:30:03.522342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.522285    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerDied","Data":"9186419f1417db947fadb18a82f0f139d1d7e4f66f388670d1c1e00f3cff4a3c"}
Nov 03 20:30:03.522616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.522598    5424 scope.go:122] "RemoveContainer" containerID="9186419f1417db947fadb18a82f0f139d1d7e4f66f388670d1c1e00f3cff4a3c"
Nov 03 20:30:03.524258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.524239    5424 generic.go:356] "Generic (PLEG): container finished" podID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerID="053b5e622d5fcef097bebc65ec1796cda1ffbfcdba4445c97294a3d409bffa0d" exitCode=60
Nov 03 20:30:03.524360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.524335    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerDied","Data":"053b5e622d5fcef097bebc65ec1796cda1ffbfcdba4445c97294a3d409bffa0d"}
Nov 03 20:30:03.524655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.524633    5424 scope.go:122] "RemoveContainer" containerID="053b5e622d5fcef097bebc65ec1796cda1ffbfcdba4445c97294a3d409bffa0d"
Nov 03 20:30:03.526129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.526107    5424 generic.go:356] "Generic (PLEG): container finished" podID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerID="2825a96849dc9e62637cfa20f7d97ff2ed1894e746f0217f28926b1a7d91dcb2" exitCode=60
Nov 03 20:30:03.526247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.526211    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerDied","Data":"2825a96849dc9e62637cfa20f7d97ff2ed1894e746f0217f28926b1a7d91dcb2"}
Nov 03 20:30:03.526427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.526411    5424 scope.go:122] "RemoveContainer" containerID="2825a96849dc9e62637cfa20f7d97ff2ed1894e746f0217f28926b1a7d91dcb2"
Nov 03 20:30:03.528020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.528001    5424 generic.go:356] "Generic (PLEG): container finished" podID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerID="550bc4cb6d469836ed73cb57d86221322ca569b95b53bb3170750fbbdebf0afa" exitCode=60
Nov 03 20:30:03.528100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.528064    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerDied","Data":"550bc4cb6d469836ed73cb57d86221322ca569b95b53bb3170750fbbdebf0afa"}
Nov 03 20:30:03.528390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.528371    5424 scope.go:122] "RemoveContainer" containerID="550bc4cb6d469836ed73cb57d86221322ca569b95b53bb3170750fbbdebf0afa"
Nov 03 20:30:03.530060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.530036    5424 generic.go:356] "Generic (PLEG): container finished" podID="ab0ad568-08e6-4811-90e3-719b385d8949" containerID="77c0e1c500e9f3b61f8d2045ca83b2b983dcdbec2ab02f82c9535f6d5d11ef9c" exitCode=60
Nov 03 20:30:03.530136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.530122    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerDied","Data":"77c0e1c500e9f3b61f8d2045ca83b2b983dcdbec2ab02f82c9535f6d5d11ef9c"}
Nov 03 20:30:03.530570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.530548    5424 scope.go:122] "RemoveContainer" containerID="77c0e1c500e9f3b61f8d2045ca83b2b983dcdbec2ab02f82c9535f6d5d11ef9c"
Nov 03 20:30:03.532580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.532509    5424 generic.go:356] "Generic (PLEG): container finished" podID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerID="a97e155be0f5cc1370f076bdcba7e76931b05c8c9269f1ec7c013ab3f2e9fa71" exitCode=60
Nov 03 20:30:03.532661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.532598    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerDied","Data":"a97e155be0f5cc1370f076bdcba7e76931b05c8c9269f1ec7c013ab3f2e9fa71"}
Nov 03 20:30:03.532962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.532943    5424 scope.go:122] "RemoveContainer" containerID="a97e155be0f5cc1370f076bdcba7e76931b05c8c9269f1ec7c013ab3f2e9fa71"
Nov 03 20:30:03.534285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.534265    5424 generic.go:356] "Generic (PLEG): container finished" podID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerID="473a51bfb94a29294c56ee92e604ac3de476a6b06afd6dbc23214c5e5595f430" exitCode=60
Nov 03 20:30:03.534384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.534338    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerDied","Data":"473a51bfb94a29294c56ee92e604ac3de476a6b06afd6dbc23214c5e5595f430"}
Nov 03 20:30:03.534623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.534599    5424 scope.go:122] "RemoveContainer" containerID="473a51bfb94a29294c56ee92e604ac3de476a6b06afd6dbc23214c5e5595f430"
Nov 03 20:30:03.536159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.536138    5424 generic.go:356] "Generic (PLEG): container finished" podID="41e66338-9228-4372-90ef-44c3e3b199f3" containerID="4a09bf397a895b2d43dc9b6354440246687a24cca8a0b7382cd702466bd1145e" exitCode=60
Nov 03 20:30:03.536271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.536246    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerDied","Data":"4a09bf397a895b2d43dc9b6354440246687a24cca8a0b7382cd702466bd1145e"}
Nov 03 20:30:03.536519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.536502    5424 scope.go:122] "RemoveContainer" containerID="4a09bf397a895b2d43dc9b6354440246687a24cca8a0b7382cd702466bd1145e"
Nov 03 20:30:03.546474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.546307    5424 generic.go:356] "Generic (PLEG): container finished" podID="cab51712-e9b0-4496-a4f7-9d366975228a" containerID="0121d4eb7d2bf9231d821458139976dfd137dc5c435da94e4008c4dbc8ee020a" exitCode=60
Nov 03 20:30:03.546474 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.546358    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerDied","Data":"0121d4eb7d2bf9231d821458139976dfd137dc5c435da94e4008c4dbc8ee020a"}
Nov 03 20:30:03.547061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.547041    5424 scope.go:122] "RemoveContainer" containerID="0121d4eb7d2bf9231d821458139976dfd137dc5c435da94e4008c4dbc8ee020a"
Nov 03 20:30:03.924953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:03.924922    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c31676-ec28-4946-97da-6551112ca316" path="/var/lib/kubelet/pods/09c31676-ec28-4946-97da-6551112ca316/volumes"
Nov 03 20:30:04.566910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.566875    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerStarted","Data":"36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"}
Nov 03 20:30:04.573522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.573481    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerStarted","Data":"c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"}
Nov 03 20:30:04.576104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.576069    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerStarted","Data":"5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6"}
Nov 03 20:30:04.578081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.578056    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerStarted","Data":"2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"}
Nov 03 20:30:04.580296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.580274    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerStarted","Data":"08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"}
Nov 03 20:30:04.582254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.582231    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerStarted","Data":"dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"}
Nov 03 20:30:04.584167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.584148    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerStarted","Data":"9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"}
Nov 03 20:30:04.586184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.586153    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerStarted","Data":"dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"}
Nov 03 20:30:04.593190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.593158    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerStarted","Data":"fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"}
Nov 03 20:30:04.595187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.595164    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerStarted","Data":"e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"}
Nov 03 20:30:04.597120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.597099    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerStarted","Data":"763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"}
Nov 03 20:30:04.599217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:04.599196    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerStarted","Data":"e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"}
Nov 03 20:30:05.773589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.758815    5424 generic.go:356] "Generic (PLEG): container finished" podID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerID="2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f" exitCode=60
Nov 03 20:30:05.773589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.758942    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerDied","Data":"2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"}
Nov 03 20:30:05.773589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.759005    5424 scope.go:122] "RemoveContainer" containerID="9186419f1417db947fadb18a82f0f139d1d7e4f66f388670d1c1e00f3cff4a3c"
Nov 03 20:30:05.773589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.759698    5424 scope.go:122] "RemoveContainer" containerID="2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"
Nov 03 20:30:05.773589 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.760221    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz_odh-ai-gateway-infra(547a2d0a-888e-48ce-8cb4-a7535fc422b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4"
Nov 03 20:30:05.813552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.813157    5424 generic.go:356] "Generic (PLEG): container finished" podID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerID="08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae" exitCode=60
Nov 03 20:30:05.813552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.813291    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerDied","Data":"08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"}
Nov 03 20:30:05.816346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.815075    5424 scope.go:122] "RemoveContainer" containerID="08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"
Nov 03 20:30:05.816346 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.815696    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7_odh-ai-gateway-infra(1adbabc7-f309-4e4c-b02d-d7c000dff668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668"
Nov 03 20:30:05.862641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.841192    5424 generic.go:356] "Generic (PLEG): container finished" podID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerID="dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0" exitCode=60
Nov 03 20:30:05.862641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.841310    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerDied","Data":"dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"}
Nov 03 20:30:05.862641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.841869    5424 scope.go:122] "RemoveContainer" containerID="dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"
Nov 03 20:30:05.862641 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.842355    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t_odh-ai-gateway-infra(51ed147e-3d13-4ec0-8b3c-c0c01098e411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411"
Nov 03 20:30:05.940957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.940098    5424 generic.go:356] "Generic (PLEG): container finished" podID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerID="9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100" exitCode=60
Nov 03 20:30:05.940957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.940693    5424 scope.go:122] "RemoveContainer" containerID="9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"
Nov 03 20:30:05.945756 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.941931    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298_odh-ai-gateway-infra(364e61c0-ddfc-482a-ba85-5d97a88fb413)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413"
Nov 03 20:30:05.957648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.946602    5424 scope.go:122] "RemoveContainer" containerID="053b5e622d5fcef097bebc65ec1796cda1ffbfcdba4445c97294a3d409bffa0d"
Nov 03 20:30:05.966334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.966302    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerDied","Data":"9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"}
Nov 03 20:30:05.966723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.966698    5424 generic.go:356] "Generic (PLEG): container finished" podID="ab0ad568-08e6-4811-90e3-719b385d8949" containerID="dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100" exitCode=60
Nov 03 20:30:05.966823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.966778    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerDied","Data":"dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"}
Nov 03 20:30:05.967166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.967141    5424 scope.go:122] "RemoveContainer" containerID="dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"
Nov 03 20:30:05.967656 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.967634    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8_odh-ai-gateway-infra(ab0ad568-08e6-4811-90e3-719b385d8949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" podUID="ab0ad568-08e6-4811-90e3-719b385d8949"
Nov 03 20:30:05.969739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.969699    5424 generic.go:356] "Generic (PLEG): container finished" podID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerID="fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70" exitCode=60
Nov 03 20:30:05.969996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.969967    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerDied","Data":"fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"}
Nov 03 20:30:05.970244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.970203    5424 scope.go:122] "RemoveContainer" containerID="fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"
Nov 03 20:30:05.970714 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.970675    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-cmkl9_odh-ai-gateway-infra(b33c35c5-a969-4d15-b7d0-e965f3100439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439"
Nov 03 20:30:05.974022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.973997    5424 generic.go:356] "Generic (PLEG): container finished" podID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerID="e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf" exitCode=60
Nov 03 20:30:05.974232 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.974083    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerDied","Data":"e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"}
Nov 03 20:30:05.974428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.974389    5424 scope.go:122] "RemoveContainer" containerID="e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"
Nov 03 20:30:05.974868 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.974844    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl_odh-ai-gateway-infra(e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4"
Nov 03 20:30:05.977062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.977040    5424 generic.go:356] "Generic (PLEG): container finished" podID="41e66338-9228-4372-90ef-44c3e3b199f3" containerID="763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6" exitCode=60
Nov 03 20:30:05.977211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.977119    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerDied","Data":"763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"}
Nov 03 20:30:05.977469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.977448    5424 scope.go:122] "RemoveContainer" containerID="763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"
Nov 03 20:30:05.977951 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.977924    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj_odh-ai-gateway-infra(41e66338-9228-4372-90ef-44c3e3b199f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" podUID="41e66338-9228-4372-90ef-44c3e3b199f3"
Nov 03 20:30:05.980366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.980339    5424 generic.go:356] "Generic (PLEG): container finished" podID="cab51712-e9b0-4496-a4f7-9d366975228a" containerID="e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e" exitCode=60
Nov 03 20:30:05.980621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.980415    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerDied","Data":"e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"}
Nov 03 20:30:05.980820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.980802    5424 scope.go:122] "RemoveContainer" containerID="e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"
Nov 03 20:30:05.981240 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.981216    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf_odh-ai-gateway-infra(cab51712-e9b0-4496-a4f7-9d366975228a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" podUID="cab51712-e9b0-4496-a4f7-9d366975228a"
Nov 03 20:30:05.983013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.982989    5424 generic.go:356] "Generic (PLEG): container finished" podID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerID="36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a" exitCode=60
Nov 03 20:30:05.983139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.983117    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerDied","Data":"36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"}
Nov 03 20:30:05.983331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.983318    5424 scope.go:122] "RemoveContainer" containerID="36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"
Nov 03 20:30:05.983626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.983603    5424 scope.go:122] "RemoveContainer" containerID="2825a96849dc9e62637cfa20f7d97ff2ed1894e746f0217f28926b1a7d91dcb2"
Nov 03 20:30:05.983680 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.983663    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh_odh-ai-gateway-infra(d5aed298-5fe8-4767-ab2a-6cb3c084150e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e"
Nov 03 20:30:05.985436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.985417    5424 generic.go:356] "Generic (PLEG): container finished" podID="617670b7-3a01-49cd-887d-4b1c6da28416" containerID="c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c" exitCode=60
Nov 03 20:30:05.985564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.985498    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerDied","Data":"c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"}
Nov 03 20:30:05.985918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.985898    5424 scope.go:122] "RemoveContainer" containerID="c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"
Nov 03 20:30:05.986245 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.986219    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42_odh-ai-gateway-infra(617670b7-3a01-49cd-887d-4b1c6da28416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" podUID="617670b7-3a01-49cd-887d-4b1c6da28416"
Nov 03 20:30:05.987605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.987575    5424 generic.go:356] "Generic (PLEG): container finished" podID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerID="5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6" exitCode=60
Nov 03 20:30:05.987704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.987654    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerDied","Data":"5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6"}
Nov 03 20:30:05.987945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.987890    5424 scope.go:122] "RemoveContainer" containerID="5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6"
Nov 03 20:30:05.988227 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:05.988206    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc_odh-ai-gateway-infra(422fb73b-5613-4f3d-bebc-2fcbaabccc6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e"
Nov 03 20:30:05.998997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:05.998980    5424 scope.go:122] "RemoveContainer" containerID="550bc4cb6d469836ed73cb57d86221322ca569b95b53bb3170750fbbdebf0afa"
Nov 03 20:30:06.016799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.016776    5424 scope.go:122] "RemoveContainer" containerID="77c0e1c500e9f3b61f8d2045ca83b2b983dcdbec2ab02f82c9535f6d5d11ef9c"
Nov 03 20:30:06.037761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.037729    5424 scope.go:122] "RemoveContainer" containerID="a97e155be0f5cc1370f076bdcba7e76931b05c8c9269f1ec7c013ab3f2e9fa71"
Nov 03 20:30:06.060950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.060923    5424 scope.go:122] "RemoveContainer" containerID="473a51bfb94a29294c56ee92e604ac3de476a6b06afd6dbc23214c5e5595f430"
Nov 03 20:30:06.080477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.080444    5424 scope.go:122] "RemoveContainer" containerID="4a09bf397a895b2d43dc9b6354440246687a24cca8a0b7382cd702466bd1145e"
Nov 03 20:30:06.098715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.098691    5424 scope.go:122] "RemoveContainer" containerID="0121d4eb7d2bf9231d821458139976dfd137dc5c435da94e4008c4dbc8ee020a"
Nov 03 20:30:06.117696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.117672    5424 scope.go:122] "RemoveContainer" containerID="512cbe6534356be9679dba2099af413907bc7466096dbfdcfc1bdf785434b4d3"
Nov 03 20:30:06.134064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.134041    5424 scope.go:122] "RemoveContainer" containerID="bf8a25b31898541b6fca63de43f9b91fc3538ea90ba1de4884436c014f2c393d"
Nov 03 20:30:06.149559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:06.149522    5424 scope.go:122] "RemoveContainer" containerID="cb9281c68c79759f7c57a483ee90366fa0855a85fc7d752d386bc188f4a9667e"
Nov 03 20:30:07.000721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.000692    5424 scope.go:122] "RemoveContainer" containerID="2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"
Nov 03 20:30:07.001790 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.001756    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz_odh-ai-gateway-infra(547a2d0a-888e-48ce-8cb4-a7535fc422b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4"
Nov 03 20:30:07.002817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.002789    5424 scope.go:122] "RemoveContainer" containerID="08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"
Nov 03 20:30:07.003295 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.003271    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7_odh-ai-gateway-infra(1adbabc7-f309-4e4c-b02d-d7c000dff668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668"
Nov 03 20:30:07.005109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.005073    5424 scope.go:122] "RemoveContainer" containerID="dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"
Nov 03 20:30:07.005595 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.005566    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t_odh-ai-gateway-infra(51ed147e-3d13-4ec0-8b3c-c0c01098e411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411"
Nov 03 20:30:07.007205 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.007178    5424 scope.go:122] "RemoveContainer" containerID="9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"
Nov 03 20:30:07.009513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.009484    5424 scope.go:122] "RemoveContainer" containerID="dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"
Nov 03 20:30:07.009699 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.009672    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298_odh-ai-gateway-infra(364e61c0-ddfc-482a-ba85-5d97a88fb413)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413"
Nov 03 20:30:07.010035 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.010007    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8_odh-ai-gateway-infra(ab0ad568-08e6-4811-90e3-719b385d8949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" podUID="ab0ad568-08e6-4811-90e3-719b385d8949"
Nov 03 20:30:07.011908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.011886    5424 scope.go:122] "RemoveContainer" containerID="fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"
Nov 03 20:30:07.012380 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.012350    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-cmkl9_odh-ai-gateway-infra(b33c35c5-a969-4d15-b7d0-e965f3100439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439"
Nov 03 20:30:07.013975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.013954    5424 scope.go:122] "RemoveContainer" containerID="e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"
Nov 03 20:30:07.014394 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.014368    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl_odh-ai-gateway-infra(e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4"
Nov 03 20:30:07.015836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.015816    5424 scope.go:122] "RemoveContainer" containerID="763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"
Nov 03 20:30:07.016178 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.016154    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj_odh-ai-gateway-infra(41e66338-9228-4372-90ef-44c3e3b199f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" podUID="41e66338-9228-4372-90ef-44c3e3b199f3"
Nov 03 20:30:07.017700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.017684    5424 scope.go:122] "RemoveContainer" containerID="e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"
Nov 03 20:30:07.018026 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.018005    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf_odh-ai-gateway-infra(cab51712-e9b0-4496-a4f7-9d366975228a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" podUID="cab51712-e9b0-4496-a4f7-9d366975228a"
Nov 03 20:30:07.019647 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.019627    5424 scope.go:122] "RemoveContainer" containerID="36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"
Nov 03 20:30:07.019932 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.019907    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh_odh-ai-gateway-infra(d5aed298-5fe8-4767-ab2a-6cb3c084150e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e"
Nov 03 20:30:07.021321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.021302    5424 scope.go:122] "RemoveContainer" containerID="c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"
Nov 03 20:30:07.021640 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.021619    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42_odh-ai-gateway-infra(617670b7-3a01-49cd-887d-4b1c6da28416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" podUID="617670b7-3a01-49cd-887d-4b1c6da28416"
Nov 03 20:30:07.022992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:07.022976    5424 scope.go:122] "RemoveContainer" containerID="5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6"
Nov 03 20:30:07.023205 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:07.023189    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc_odh-ai-gateway-infra(422fb73b-5613-4f3d-bebc-2fcbaabccc6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e"
Nov 03 20:30:12.294472 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:12.294414    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:30:14.912972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:14.912942    5424 scope.go:122] "RemoveContainer" containerID="2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"
Nov 03 20:30:15.913631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:15.913585    5424 scope.go:122] "RemoveContainer" containerID="c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"
Nov 03 20:30:15.915045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:15.914778    5424 scope.go:122] "RemoveContainer" containerID="e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"
Nov 03 20:30:16.111767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.111729    5424 generic.go:356] "Generic (PLEG): container finished" podID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerID="e5da32bf9aac183f37e4ca22c246a32f0881ce5c0c7937fc4bf41e8fe7c2946e" exitCode=60
Nov 03 20:30:16.113972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.112075    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerDied","Data":"e5da32bf9aac183f37e4ca22c246a32f0881ce5c0c7937fc4bf41e8fe7c2946e"}
Nov 03 20:30:16.113972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.112150    5424 scope.go:122] "RemoveContainer" containerID="2cee11a1f74204399ae2c1322c33a38733614f80edb0666a47689fea768df50f"
Nov 03 20:30:16.113972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.112893    5424 scope.go:122] "RemoveContainer" containerID="e5da32bf9aac183f37e4ca22c246a32f0881ce5c0c7937fc4bf41e8fe7c2946e"
Nov 03 20:30:16.113972 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:16.113490    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz_odh-ai-gateway-infra(547a2d0a-888e-48ce-8cb4-a7535fc422b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4"
Nov 03 20:30:16.913620 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.913523    5424 scope.go:122] "RemoveContainer" containerID="dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"
Nov 03 20:30:16.913780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.913664    5424 scope.go:122] "RemoveContainer" containerID="dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"
Nov 03 20:30:16.914046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.913846    5424 scope.go:122] "RemoveContainer" containerID="fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"
Nov 03 20:30:16.914094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:16.914056    5424 scope.go:122] "RemoveContainer" containerID="08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"
Nov 03 20:30:17.130162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.130131    5424 generic.go:356] "Generic (PLEG): container finished" podID="617670b7-3a01-49cd-887d-4b1c6da28416" containerID="6682552e91af848189340be7c6bfd4807c54e3eff612590bb15e581322e27b35" exitCode=60
Nov 03 20:30:17.130309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.130203    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerDied","Data":"6682552e91af848189340be7c6bfd4807c54e3eff612590bb15e581322e27b35"}
Nov 03 20:30:17.130309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.130268    5424 scope.go:122] "RemoveContainer" containerID="c46d6d9b9caec1d996290f0329155033fa339a061b3a4bd21fb7c9fdb429c93c"
Nov 03 20:30:17.130754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.130733    5424 scope.go:122] "RemoveContainer" containerID="6682552e91af848189340be7c6bfd4807c54e3eff612590bb15e581322e27b35"
Nov 03 20:30:17.131085 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:17.131054    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42_odh-ai-gateway-infra(617670b7-3a01-49cd-887d-4b1c6da28416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" podUID="617670b7-3a01-49cd-887d-4b1c6da28416"
Nov 03 20:30:17.134392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.134370    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerStarted","Data":"2d594401ff4f5c0abad090eb279e9aee7948e3140b474cd7170e59128607143b"}
Nov 03 20:30:17.134764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.134749    5424 scope.go:122] "RemoveContainer" containerID="2d594401ff4f5c0abad090eb279e9aee7948e3140b474cd7170e59128607143b"
Nov 03 20:30:17.135031 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:17.135015    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl_odh-ai-gateway-infra(e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4"
Nov 03 20:30:17.188210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.188123    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:17.414419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.414392    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:17.497912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.497883    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\" (UniqueName: \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\") pod \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") "
Nov 03 20:30:17.498099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.497925    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\") pod \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\" (UID: \"547a2d0a-888e-48ce-8cb4-a7535fc422b4\") "
Nov 03 20:30:17.498660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.498604    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca" pod "547a2d0a-888e-48ce-8cb4-a7535fc422b4" (UID: "547a2d0a-888e-48ce-8cb4-a7535fc422b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:17.499069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.499041    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547a2d0a-888e-48ce-8cb4-a7535fc422b4-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:17.502216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.502180    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk" pod "547a2d0a-888e-48ce-8cb4-a7535fc422b4" (UID: "547a2d0a-888e-48ce-8cb4-a7535fc422b4"). InnerVolumeSpecName "kube-api-access-6knzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:17.560468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.560438    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:17.604092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.604045    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knzk\" (UniqueName: \"kubernetes.io/projected/547a2d0a-888e-48ce-8cb4-a7535fc422b4-kube-api-access-6knzk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:17.993895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.990610    5424 scope.go:122] "RemoveContainer" containerID="763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"
Nov 03 20:30:17.995812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:17.995786    5424 scope.go:122] "RemoveContainer" containerID="e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"
Nov 03 20:30:18.488805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.486442    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518411    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"]
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518450    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518496    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb87f429-bbxht"]
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518520    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518575    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz" event={"ID":"547a2d0a-888e-48ce-8cb4-a7535fc422b4","Type":"ContainerDied","Data":"cd3249b6d03acc94c54eb0c59531844508b47b9fb45b6fe1b51074e75dcc0128"}
Nov 03 20:30:18.524576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.518610    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerStarted","Data":"e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d"}
Nov 03 20:30:18.549081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.543660    5424 scope.go:122] "RemoveContainer" containerID="e5da32bf9aac183f37e4ca22c246a32f0881ce5c0c7937fc4bf41e8fe7c2946e"
Nov 03 20:30:18.561621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.561583    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerStarted","Data":"88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b"}
Nov 03 20:30:18.562055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.562033    5424 scope.go:122] "RemoveContainer" containerID="88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b"
Nov 03 20:30:18.562485 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:18.562457    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t_odh-ai-gateway-infra(51ed147e-3d13-4ec0-8b3c-c0c01098e411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411"
Nov 03 20:30:18.570196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.570166    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerStarted","Data":"534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445"}
Nov 03 20:30:18.570845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.570819    5424 scope.go:122] "RemoveContainer" containerID="534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445"
Nov 03 20:30:18.571342 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:18.571310    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8_odh-ai-gateway-infra(ab0ad568-08e6-4811-90e3-719b385d8949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" podUID="ab0ad568-08e6-4811-90e3-719b385d8949"
Nov 03 20:30:18.577924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.577896    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerStarted","Data":"8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8"}
Nov 03 20:30:18.578608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.578589    5424 scope.go:122] "RemoveContainer" containerID="8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8"
Nov 03 20:30:18.588234 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:18.588203    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-cmkl9_odh-ai-gateway-infra(b33c35c5-a969-4d15-b7d0-e965f3100439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439"
Nov 03 20:30:18.595274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.594070    5424 generic.go:356] "Generic (PLEG): container finished" podID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerID="2d594401ff4f5c0abad090eb279e9aee7948e3140b474cd7170e59128607143b" exitCode=60
Nov 03 20:30:18.595274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.594271    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerDied","Data":"2d594401ff4f5c0abad090eb279e9aee7948e3140b474cd7170e59128607143b"}
Nov 03 20:30:18.595274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.594344    5424 scope.go:122] "RemoveContainer" containerID="e4fa1389505b3d71318667128738b66f87aa3c0aab49e15b6dac409227be0ddf"
Nov 03 20:30:18.654855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.654827    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:18.658726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.658696    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\") pod \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") "
Nov 03 20:30:18.659031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.659009    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\" (UniqueName: \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\") pod \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\" (UID: \"422fb73b-5613-4f3d-bebc-2fcbaabccc6e\") "
Nov 03 20:30:18.663102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.661038    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca" pod "422fb73b-5613-4f3d-bebc-2fcbaabccc6e" (UID: "422fb73b-5613-4f3d-bebc-2fcbaabccc6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.734758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.734720    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x" pod "422fb73b-5613-4f3d-bebc-2fcbaabccc6e" (UID: "422fb73b-5613-4f3d-bebc-2fcbaabccc6e"). InnerVolumeSpecName "kube-api-access-zmv4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:18.764521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.764441    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:18.764807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.764786    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmv4x\" (UniqueName: \"kubernetes.io/projected/422fb73b-5613-4f3d-bebc-2fcbaabccc6e-kube-api-access-zmv4x\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:18.891579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.891549    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:18.910782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.910755    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:18.913452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.913433    5424 scope.go:122] "RemoveContainer" containerID="36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"
Nov 03 20:30:18.913941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:18.913916    5424 scope.go:122] "RemoveContainer" containerID="9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"
Nov 03 20:30:19.044203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.044173    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:19.050452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.050430    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-76b894-29895630-gj9gz"]
Nov 03 20:30:19.075492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.075462    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\") pod \"617670b7-3a01-49cd-887d-4b1c6da28416\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") "
Nov 03 20:30:19.075695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.075670    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\" (UniqueName: \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\") pod \"617670b7-3a01-49cd-887d-4b1c6da28416\" (UID: \"617670b7-3a01-49cd-887d-4b1c6da28416\") "
Nov 03 20:30:19.075777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.075707    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\" (UniqueName: \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\") pod \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") "
Nov 03 20:30:19.075924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.075903    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\") pod \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\" (UID: \"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4\") "
Nov 03 20:30:19.076040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.075906    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca" pod "617670b7-3a01-49cd-887d-4b1c6da28416" (UID: "617670b7-3a01-49cd-887d-4b1c6da28416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:19.076460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.076436    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca" pod "e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" (UID: "e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:19.076911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.076882    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.077018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.076917    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617670b7-3a01-49cd-887d-4b1c6da28416-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.078807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.078779    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d" pod "617670b7-3a01-49cd-887d-4b1c6da28416" (UID: "617670b7-3a01-49cd-887d-4b1c6da28416"). InnerVolumeSpecName "kube-api-access-dc76d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.079193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.079162    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59" pod "e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" (UID: "e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4"). InnerVolumeSpecName "kube-api-access-prg59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.129450 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:19.129403    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:19.178627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.178592    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc76d\" (UniqueName: \"kubernetes.io/projected/617670b7-3a01-49cd-887d-4b1c6da28416-kube-api-access-dc76d\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.178627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.178620    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prg59\" (UniqueName: \"kubernetes.io/projected/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4-kube-api-access-prg59\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.608074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.607964    5424 generic.go:356] "Generic (PLEG): container finished" podID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerID="e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d" exitCode=60
Nov 03 20:30:19.608074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.608034    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerDied","Data":"e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d"}
Nov 03 20:30:19.608291 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.608105    5424 scope.go:122] "RemoveContainer" containerID="08adf8e3fec0767f188ad85a6dda51206763b85b0c148a21be326132969848ae"
Nov 03 20:30:19.608611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.608590    5424 scope.go:122] "RemoveContainer" containerID="e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d"
Nov 03 20:30:19.609093 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:19.609067    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7_odh-ai-gateway-infra(1adbabc7-f309-4e4c-b02d-d7c000dff668)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668"
Nov 03 20:30:19.610439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.610415    5424 generic.go:356] "Generic (PLEG): container finished" podID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerID="88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b" exitCode=60
Nov 03 20:30:19.610603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.610492    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerDied","Data":"88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b"}
Nov 03 20:30:19.610999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.610980    5424 scope.go:122] "RemoveContainer" containerID="88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b"
Nov 03 20:30:19.611443 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:19.611423    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t_odh-ai-gateway-infra(51ed147e-3d13-4ec0-8b3c-c0c01098e411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411"
Nov 03 20:30:19.612624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.612603    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerStarted","Data":"dea61b4ed29557481f0d90b8dfb3fdb88db58bd50d397d99bf758998a63a56cf"}
Nov 03 20:30:19.614050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.614028    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:19.614917 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.614898    5424 generic.go:356] "Generic (PLEG): container finished" podID="ab0ad568-08e6-4811-90e3-719b385d8949" containerID="534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445" exitCode=60
Nov 03 20:30:19.615005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.614974    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerDied","Data":"534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445"}
Nov 03 20:30:19.615481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.615460    5424 scope.go:122] "RemoveContainer" containerID="534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445"
Nov 03 20:30:19.615933 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:19.615908    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8_odh-ai-gateway-infra(ab0ad568-08e6-4811-90e3-719b385d8949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" podUID="ab0ad568-08e6-4811-90e3-719b385d8949"
Nov 03 20:30:19.617014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.616994    5424 generic.go:356] "Generic (PLEG): container finished" podID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerID="8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8" exitCode=60
Nov 03 20:30:19.617123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.617068    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerDied","Data":"8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8"}
Nov 03 20:30:19.617451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.617433    5424 scope.go:122] "RemoveContainer" containerID="8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8"
Nov 03 20:30:19.617868 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:19.617843    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-cmkl9_odh-ai-gateway-infra(b33c35c5-a969-4d15-b7d0-e965f3100439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439"
Nov 03 20:30:19.627411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.627380    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"
Nov 03 20:30:19.627411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.627389    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl" event={"ID":"e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4","Type":"ContainerDied","Data":"c8deee625e73627fb4b291b23c7067a1816abb99ac2d993af83e0545afaf1b09"}
Nov 03 20:30:19.631182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.630084    5424 scope.go:122] "RemoveContainer" containerID="dc21a727bf9baf0d47ddc4664d65efe0c954e5085b927d076122aa44c139bac0"
Nov 03 20:30:19.643967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.643467    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerStarted","Data":"02f392ecc7bc90093ac8ac335bec0d4eba8184db2b1608b5c580bdb41e1845c2"}
Nov 03 20:30:19.645893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.645859    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerStarted","Data":"1cbd86fbbe1ab9dab87f473c6fffeacf183e49baf4a8516340e20acaf72b30c2"}
Nov 03 20:30:19.647953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.647927    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerStarted","Data":"7112db8031413fb55799969803e8f7d6206e345bf32988ce64e67d3aabf7a317"}
Nov 03 20:30:19.649583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.649561    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"
Nov 03 20:30:19.649688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.649604    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42" event={"ID":"617670b7-3a01-49cd-887d-4b1c6da28416","Type":"ContainerDied","Data":"e507a8a9533dd044c03ae4d680cf48b11a89371ed7b881c1b2e46da624ef1f0a"}
Nov 03 20:30:19.652266 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.652244    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc" event={"ID":"422fb73b-5613-4f3d-bebc-2fcbaabccc6e","Type":"ContainerDied","Data":"b882cd8e044283aa3905e1b8b37a268bfbca34e63f502b990b9920f89bcdd405"}
Nov 03 20:30:19.652393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.652310    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"
Nov 03 20:30:19.659689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.659668    5424 scope.go:122] "RemoveContainer" containerID="dee0a79002f7fffb419ffe95a10e06b61f53ada069d286f4ff447de2adc62100"
Nov 03 20:30:19.675137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.675117    5424 scope.go:122] "RemoveContainer" containerID="fc26cd338a396384708bdf46bc9aa5e21153b4a7a91d5d8c1ced6f5d38eb4f70"
Nov 03 20:30:19.690956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.690935    5424 scope.go:122] "RemoveContainer" containerID="2d594401ff4f5c0abad090eb279e9aee7948e3140b474cd7170e59128607143b"
Nov 03 20:30:19.702615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.702592    5424 scope.go:122] "RemoveContainer" containerID="6682552e91af848189340be7c6bfd4807c54e3eff612590bb15e581322e27b35"
Nov 03 20:30:19.714286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.714262    5424 scope.go:122] "RemoveContainer" containerID="5178cdd11647dbf93d309a1082b059c9af48d9f532ea35d6a29d2eebd32561b6"
Nov 03 20:30:19.731891 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.721559    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:19.734502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.734472    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:19.742019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.741995    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fd01ad-29895630-hj2kl"]
Nov 03 20:30:19.878626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.878579    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:19.928834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.928797    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" path="/var/lib/kubelet/pods/547a2d0a-888e-48ce-8cb4-a7535fc422b4/volumes"
Nov 03 20:30:19.929324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.929310    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" path="/var/lib/kubelet/pods/e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4/volumes"
Nov 03 20:30:19.929824 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.929810    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce" path="/var/lib/kubelet/pods/ee3c9b78-a3f1-466c-a6c6-aa8ab994dbce/volumes"
Nov 03 20:30:19.965617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:19.965584    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:20.017034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.017005    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:20.026042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.026006    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9973d9-29895630-sxx42"]
Nov 03 20:30:20.062731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.062695    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:20.067902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.067873    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-eb87f429-29895630-56trc"]
Nov 03 20:30:20.669998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.669970    5424 generic.go:356] "Generic (PLEG): container finished" podID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerID="dea61b4ed29557481f0d90b8dfb3fdb88db58bd50d397d99bf758998a63a56cf" exitCode=60
Nov 03 20:30:20.670136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.670045    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerDied","Data":"dea61b4ed29557481f0d90b8dfb3fdb88db58bd50d397d99bf758998a63a56cf"}
Nov 03 20:30:20.670199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.670143    5424 scope.go:122] "RemoveContainer" containerID="9ef2e754dfed5defd5465be62cc2188b81cfff8cb0bb037438d105cff4c5b100"
Nov 03 20:30:20.670718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.670698    5424 scope.go:122] "RemoveContainer" containerID="dea61b4ed29557481f0d90b8dfb3fdb88db58bd50d397d99bf758998a63a56cf"
Nov 03 20:30:20.671153 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:20.671129    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298_odh-ai-gateway-infra(364e61c0-ddfc-482a-ba85-5d97a88fb413)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413"
Nov 03 20:30:20.681048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.678928    5424 generic.go:356] "Generic (PLEG): container finished" podID="41e66338-9228-4372-90ef-44c3e3b199f3" containerID="02f392ecc7bc90093ac8ac335bec0d4eba8184db2b1608b5c580bdb41e1845c2" exitCode=60
Nov 03 20:30:20.681048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.678976    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerDied","Data":"02f392ecc7bc90093ac8ac335bec0d4eba8184db2b1608b5c580bdb41e1845c2"}
Nov 03 20:30:20.681048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.679500    5424 scope.go:122] "RemoveContainer" containerID="02f392ecc7bc90093ac8ac335bec0d4eba8184db2b1608b5c580bdb41e1845c2"
Nov 03 20:30:20.681048 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:20.680046    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj_odh-ai-gateway-infra(41e66338-9228-4372-90ef-44c3e3b199f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" podUID="41e66338-9228-4372-90ef-44c3e3b199f3"
Nov 03 20:30:20.681827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.681803    5424 generic.go:356] "Generic (PLEG): container finished" podID="cab51712-e9b0-4496-a4f7-9d366975228a" containerID="1cbd86fbbe1ab9dab87f473c6fffeacf183e49baf4a8516340e20acaf72b30c2" exitCode=60
Nov 03 20:30:20.681979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.681961    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerDied","Data":"1cbd86fbbe1ab9dab87f473c6fffeacf183e49baf4a8516340e20acaf72b30c2"}
Nov 03 20:30:20.682502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.682468    5424 scope.go:122] "RemoveContainer" containerID="1cbd86fbbe1ab9dab87f473c6fffeacf183e49baf4a8516340e20acaf72b30c2"
Nov 03 20:30:20.684406 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:20.684378    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf_odh-ai-gateway-infra(cab51712-e9b0-4496-a4f7-9d366975228a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" podUID="cab51712-e9b0-4496-a4f7-9d366975228a"
Nov 03 20:30:20.685314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.685294    5424 generic.go:356] "Generic (PLEG): container finished" podID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerID="7112db8031413fb55799969803e8f7d6206e345bf32988ce64e67d3aabf7a317" exitCode=60
Nov 03 20:30:20.685431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.685413    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerDied","Data":"7112db8031413fb55799969803e8f7d6206e345bf32988ce64e67d3aabf7a317"}
Nov 03 20:30:20.685868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.685850    5424 scope.go:122] "RemoveContainer" containerID="7112db8031413fb55799969803e8f7d6206e345bf32988ce64e67d3aabf7a317"
Nov 03 20:30:20.686261 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:20.686240    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh_odh-ai-gateway-infra(d5aed298-5fe8-4767-ab2a-6cb3c084150e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e"
Nov 03 20:30:20.689247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.689228    5424 scope.go:122] "RemoveContainer" containerID="763b28605e96e6e6ac8bcd2ccdffad92eec943cd3bda1d845a78a33079cab6a6"
Nov 03 20:30:20.717698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.717668    5424 scope.go:122] "RemoveContainer" containerID="e518600211add97d222fdc428bd5724381a57586eb38cd656ea436e632029a8e"
Nov 03 20:30:20.742726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.742561    5424 scope.go:122] "RemoveContainer" containerID="2497fbf248ec83b0681f2cb6979b7e6e46ffcba19aef5fcdae31d82020880e6b"
Nov 03 20:30:20.748060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.748028    5424 scope.go:122] "RemoveContainer" containerID="36d7121e4ef29df65992d167f46d7da563fb375cb6cff024aed8b3e54f95240a"
Nov 03 20:30:20.763058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.763032    5424 scope.go:122] "RemoveContainer" containerID="4ed88b39b7c33babd996028d17ff093e43b600fb5378e53e260fede058222de7"
Nov 03 20:30:20.790733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.790707    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:20.805999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.805971    5424 scope.go:122] "RemoveContainer" containerID="ddef6513cd4d4df8428efdcaf87c805c592cc51a1f68294eac1a7af1230dfb6a"
Nov 03 20:30:20.879578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.879496    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:20.955112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:20.955082    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:21.015382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.014775    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:21.022590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.022520    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:21.024935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.024909    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:21.078065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.078032    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:21.083943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.083916    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:21.130377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130343    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\") pod \"ab0ad568-08e6-4811-90e3-719b385d8949\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") "
Nov 03 20:30:21.130544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130393    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\" (UniqueName: \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\") pod \"b33c35c5-a969-4d15-b7d0-e965f3100439\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") "
Nov 03 20:30:21.130544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130449    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\" (UniqueName: \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\") pod \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") "
Nov 03 20:30:21.130675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130542    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\" (UniqueName: \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\") pod \"ab0ad568-08e6-4811-90e3-719b385d8949\" (UID: \"ab0ad568-08e6-4811-90e3-719b385d8949\") "
Nov 03 20:30:21.130675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130591    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\") pod \"1adbabc7-f309-4e4c-b02d-d7c000dff668\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") "
Nov 03 20:30:21.130784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130765    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\" (UniqueName: \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\") pod \"1adbabc7-f309-4e4c-b02d-d7c000dff668\" (UID: \"1adbabc7-f309-4e4c-b02d-d7c000dff668\") "
Nov 03 20:30:21.130942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130926    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\") pod \"b33c35c5-a969-4d15-b7d0-e965f3100439\" (UID: \"b33c35c5-a969-4d15-b7d0-e965f3100439\") "
Nov 03 20:30:21.131003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.130974    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\") pod \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\" (UID: \"51ed147e-3d13-4ec0-8b3c-c0c01098e411\") "
Nov 03 20:30:21.133306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.133040    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca" pod "1adbabc7-f309-4e4c-b02d-d7c000dff668" (UID: "1adbabc7-f309-4e4c-b02d-d7c000dff668"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:21.133932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.133901    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca" pod "b33c35c5-a969-4d15-b7d0-e965f3100439" (UID: "b33c35c5-a969-4d15-b7d0-e965f3100439"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:21.134489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.134464    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca" pod "51ed147e-3d13-4ec0-8b3c-c0c01098e411" (UID: "51ed147e-3d13-4ec0-8b3c-c0c01098e411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:21.134846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.134824    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca" pod "ab0ad568-08e6-4811-90e3-719b385d8949" (UID: "ab0ad568-08e6-4811-90e3-719b385d8949"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:21.137714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.137687    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h" pod "ab0ad568-08e6-4811-90e3-719b385d8949" (UID: "ab0ad568-08e6-4811-90e3-719b385d8949"). InnerVolumeSpecName "kube-api-access-42g5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:21.140523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.140472    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc" pod "51ed147e-3d13-4ec0-8b3c-c0c01098e411" (UID: "51ed147e-3d13-4ec0-8b3c-c0c01098e411"). InnerVolumeSpecName "kube-api-access-dzvdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:21.140686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.140649    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv" pod "b33c35c5-a969-4d15-b7d0-e965f3100439" (UID: "b33c35c5-a969-4d15-b7d0-e965f3100439"). InnerVolumeSpecName "kube-api-access-tltfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:21.141074 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.141056    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x" pod "1adbabc7-f309-4e4c-b02d-d7c000dff668" (UID: "1adbabc7-f309-4e4c-b02d-d7c000dff668"). InnerVolumeSpecName "kube-api-access-g894x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:21.237546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237480    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g894x\" (UniqueName: \"kubernetes.io/projected/1adbabc7-f309-4e4c-b02d-d7c000dff668-kube-api-access-g894x\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237553    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33c35c5-a969-4d15-b7d0-e965f3100439-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237574    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ed147e-3d13-4ec0-8b3c-c0c01098e411-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237590    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab0ad568-08e6-4811-90e3-719b385d8949-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237605    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tltfv\" (UniqueName: \"kubernetes.io/projected/b33c35c5-a969-4d15-b7d0-e965f3100439-kube-api-access-tltfv\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237618    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvdc\" (UniqueName: \"kubernetes.io/projected/51ed147e-3d13-4ec0-8b3c-c0c01098e411-kube-api-access-dzvdc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237631    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42g5h\" (UniqueName: \"kubernetes.io/projected/ab0ad568-08e6-4811-90e3-719b385d8949-kube-api-access-42g5h\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.237756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.237644    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1adbabc7-f309-4e4c-b02d-d7c000dff668-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.704197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.704166    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7" event={"ID":"1adbabc7-f309-4e4c-b02d-d7c000dff668","Type":"ContainerDied","Data":"9859c9db3a16109ca2ee1a3114bbf079ab0ec61be94383267657b53e47d8ba1e"}
Nov 03 20:30:21.704431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.704411    5424 scope.go:122] "RemoveContainer" containerID="e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d"
Nov 03 20:30:21.704583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.704504    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"
Nov 03 20:30:21.706517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.706441    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t" event={"ID":"51ed147e-3d13-4ec0-8b3c-c0c01098e411","Type":"ContainerDied","Data":"146b56a3874cb5ed012157fc77713462c090af45eae6096aaab15ecaa0c66de8"}
Nov 03 20:30:21.706517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.706471    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"
Nov 03 20:30:21.719556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.716622    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8" event={"ID":"ab0ad568-08e6-4811-90e3-719b385d8949","Type":"ContainerDied","Data":"8370ed192e513b00297e2daf2c33bb37dfc5720c23f4a778619d3468d33423be"}
Nov 03 20:30:21.719556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.716743    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"
Nov 03 20:30:21.725999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.725773    5424 scope.go:122] "RemoveContainer" containerID="88557a0a1f0424fc80efc1d989487dfbadcd663b05fb9f8cb91972c37bd8075b"
Nov 03 20:30:21.730411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.730356    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9" event={"ID":"b33c35c5-a969-4d15-b7d0-e965f3100439","Type":"ContainerDied","Data":"1dbca3b65bb6481558c0fa0d8a7901dade98a2dcb6a569bc3de9112352b334f2"}
Nov 03 20:30:21.730611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.730497    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"
Nov 03 20:30:21.767683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.767613    5424 scope.go:122] "RemoveContainer" containerID="534a17a08d133a7d8e6a0dc7f96a76ba0536f015543120e381200bbc40810445"
Nov 03 20:30:21.782967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.782942    5424 scope.go:122] "RemoveContainer" containerID="8b9fd1e6fd8b4db89dc1a3bdafd1573b19f28efae7d95c105518e576224e05d8"
Nov 03 20:30:21.934157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.934107    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:21.943213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.943184    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" path="/var/lib/kubelet/pods/422fb73b-5613-4f3d-bebc-2fcbaabccc6e/volumes"
Nov 03 20:30:21.943971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.943949    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" path="/var/lib/kubelet/pods/617670b7-3a01-49cd-887d-4b1c6da28416/volumes"
Nov 03 20:30:21.991762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.991733    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:21.996901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.996875    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-503c4e-29895630-ms98t"]
Nov 03 20:30:21.998380 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:21.998356    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:22.019302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.019279    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:22.023497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.023475    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:22.029492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.029466    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4a71d-29895630-65kj7"]
Nov 03 20:30:22.033158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.033138    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:22.055120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055096    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\") pod \"41e66338-9228-4372-90ef-44c3e3b199f3\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") "
Nov 03 20:30:22.055355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055336    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\" (UniqueName: \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\") pod \"cab51712-e9b0-4496-a4f7-9d366975228a\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") "
Nov 03 20:30:22.055414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055395    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca" pod "41e66338-9228-4372-90ef-44c3e3b199f3" (UID: "41e66338-9228-4372-90ef-44c3e3b199f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:22.055476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055463    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\") pod \"cab51712-e9b0-4496-a4f7-9d366975228a\" (UID: \"cab51712-e9b0-4496-a4f7-9d366975228a\") "
Nov 03 20:30:22.055612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055593    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\" (UniqueName: \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\") pod \"41e66338-9228-4372-90ef-44c3e3b199f3\" (UID: \"41e66338-9228-4372-90ef-44c3e3b199f3\") "
Nov 03 20:30:22.055882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.055860    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca" pod "cab51712-e9b0-4496-a4f7-9d366975228a" (UID: "cab51712-e9b0-4496-a4f7-9d366975228a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:22.056613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.056593    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab51712-e9b0-4496-a4f7-9d366975228a-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.056700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.056616    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e66338-9228-4372-90ef-44c3e3b199f3-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.057914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.057894    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5" pod "cab51712-e9b0-4496-a4f7-9d366975228a" (UID: "cab51712-e9b0-4496-a4f7-9d366975228a"). InnerVolumeSpecName "kube-api-access-gv2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:22.058067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.058045    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6" pod "41e66338-9228-4372-90ef-44c3e3b199f3" (UID: "41e66338-9228-4372-90ef-44c3e3b199f3"). InnerVolumeSpecName "kube-api-access-fcsm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:22.059234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.059212    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:22.064738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.064719    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-cmkl9"]
Nov 03 20:30:22.154935 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.154903    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:22.155452 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:22.154970    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.157508    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\") pod \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") "
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.158176    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca" pod "d5aed298-5fe8-4767-ab2a-6cb3c084150e" (UID: "d5aed298-5fe8-4767-ab2a-6cb3c084150e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.159605    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\" (UniqueName: \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\") pod \"364e61c0-ddfc-482a-ba85-5d97a88fb413\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") "
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.159804    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\" (UniqueName: \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\") pod \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\" (UID: \"d5aed298-5fe8-4767-ab2a-6cb3c084150e\") "
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.159852    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\") pod \"364e61c0-ddfc-482a-ba85-5d97a88fb413\" (UID: \"364e61c0-ddfc-482a-ba85-5d97a88fb413\") "
Nov 03 20:30:22.160593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.160293    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca" pod "364e61c0-ddfc-482a-ba85-5d97a88fb413" (UID: "364e61c0-ddfc-482a-ba85-5d97a88fb413"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:22.161028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.160626    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv2l5\" (UniqueName: \"kubernetes.io/projected/cab51712-e9b0-4496-a4f7-9d366975228a-kube-api-access-gv2l5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.161028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.160649    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364e61c0-ddfc-482a-ba85-5d97a88fb413-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.161028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.160664    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcsm6\" (UniqueName: \"kubernetes.io/projected/41e66338-9228-4372-90ef-44c3e3b199f3-kube-api-access-fcsm6\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.161028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.160677    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aed298-5fe8-4767-ab2a-6cb3c084150e-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.162832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.162806    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-de82c8-29895630-qnxp8"]
Nov 03 20:30:22.163276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.163250    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx" pod "d5aed298-5fe8-4767-ab2a-6cb3c084150e" (UID: "d5aed298-5fe8-4767-ab2a-6cb3c084150e"). InnerVolumeSpecName "kube-api-access-ltdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:22.167361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.167334    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l" pod "364e61c0-ddfc-482a-ba85-5d97a88fb413" (UID: "364e61c0-ddfc-482a-ba85-5d97a88fb413"). InnerVolumeSpecName "kube-api-access-fcs2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:22.262705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.262424    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcs2l\" (UniqueName: \"kubernetes.io/projected/364e61c0-ddfc-482a-ba85-5d97a88fb413-kube-api-access-fcs2l\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.262961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.262933    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdgx\" (UniqueName: \"kubernetes.io/projected/d5aed298-5fe8-4767-ab2a-6cb3c084150e-kube-api-access-ltdgx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.739870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.739836    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"
Nov 03 20:30:22.739870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.739864    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf" event={"ID":"cab51712-e9b0-4496-a4f7-9d366975228a","Type":"ContainerDied","Data":"8bb91b531903016a2e123a87e9ea6fa78dc9013e6ecbcd7ab11fe5c8be08b80c"}
Nov 03 20:30:22.740123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.739931    5424 scope.go:122] "RemoveContainer" containerID="1cbd86fbbe1ab9dab87f473c6fffeacf183e49baf4a8516340e20acaf72b30c2"
Nov 03 20:30:22.741599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.741575    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"
Nov 03 20:30:22.741743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.741571    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh" event={"ID":"d5aed298-5fe8-4767-ab2a-6cb3c084150e","Type":"ContainerDied","Data":"bbddb691ec881032416cacde5e239d8807a674e078dba90e68b0aacbe4e8f544"}
Nov 03 20:30:22.747034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.747014    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"
Nov 03 20:30:22.747143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.747016    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298" event={"ID":"364e61c0-ddfc-482a-ba85-5d97a88fb413","Type":"ContainerDied","Data":"f30d65ad2272b30b60ac79fd56f86cdb4730605a7b2e298212e5c44cded79473"}
Nov 03 20:30:22.752605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.752583    5424 scope.go:122] "RemoveContainer" containerID="7112db8031413fb55799969803e8f7d6206e345bf32988ce64e67d3aabf7a317"
Nov 03 20:30:22.753908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.753664    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj" event={"ID":"41e66338-9228-4372-90ef-44c3e3b199f3","Type":"ContainerDied","Data":"69856c4a47806a60a13498d57e397bfaa2673bdaad88d2ece1e9ebf6d81e7ab1"}
Nov 03 20:30:22.753908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.753702    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"
Nov 03 20:30:22.764632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.764609    5424 scope.go:122] "RemoveContainer" containerID="dea61b4ed29557481f0d90b8dfb3fdb88db58bd50d397d99bf758998a63a56cf"
Nov 03 20:30:22.775357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.775341    5424 scope.go:122] "RemoveContainer" containerID="02f392ecc7bc90093ac8ac335bec0d4eba8184db2b1608b5c580bdb41e1845c2"
Nov 03 20:30:22.809844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.809821    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:22.820609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.820585    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e80203-29895630-hcllf"]
Nov 03 20:30:22.865962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.865882    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:22.872554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.872511    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-fab871-29895630-ljhqh"]
Nov 03 20:30:22.907114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.907090    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:22.912452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.912429    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a0866c-29895630-bd298"]
Nov 03 20:30:22.952873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.952842    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:22.957008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:22.956988    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4cdbeb-29895630-84drj"]
Nov 03 20:30:23.661803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.661766    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:30:23.925452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.925366    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" path="/var/lib/kubelet/pods/1adbabc7-f309-4e4c-b02d-d7c000dff668/volumes"
Nov 03 20:30:23.926207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.926168    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" path="/var/lib/kubelet/pods/364e61c0-ddfc-482a-ba85-5d97a88fb413/volumes"
Nov 03 20:30:23.926668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.926651    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" path="/var/lib/kubelet/pods/41e66338-9228-4372-90ef-44c3e3b199f3/volumes"
Nov 03 20:30:23.927492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.927360    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" path="/var/lib/kubelet/pods/51ed147e-3d13-4ec0-8b3c-c0c01098e411/volumes"
Nov 03 20:30:23.927844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.927826    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" path="/var/lib/kubelet/pods/ab0ad568-08e6-4811-90e3-719b385d8949/volumes"
Nov 03 20:30:23.928238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.928225    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" path="/var/lib/kubelet/pods/b33c35c5-a969-4d15-b7d0-e965f3100439/volumes"
Nov 03 20:30:23.928949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.928934    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" path="/var/lib/kubelet/pods/cab51712-e9b0-4496-a4f7-9d366975228a/volumes"
Nov 03 20:30:23.929322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:23.929309    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" path="/var/lib/kubelet/pods/d5aed298-5fe8-4767-ab2a-6cb3c084150e/volumes"
Nov 03 20:30:29.217808 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:29.217773    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:34.016674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.016630    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:30:34.018424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018393    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.018424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018424    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018441    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018450    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018464    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018471    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018484    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018492    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018502    5424 cpu_manager.go:414] "containerMap: removing container" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018509    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018521    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018546    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018556    5424 cpu_manager.go:414] "containerMap: removing container" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018566    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018581    5424 cpu_manager.go:414] "containerMap: removing container" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018588    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018605    5424 cpu_manager.go:414] "containerMap: removing container" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018612    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018627    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018633    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018642    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018649    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018663    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018670    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018688    5424 cpu_manager.go:414] "containerMap: removing container" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018695    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.018695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018708    5424 cpu_manager.go:414] "containerMap: removing container" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018716    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018731    5424 cpu_manager.go:414] "containerMap: removing container" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018738    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018756    5424 cpu_manager.go:414] "containerMap: removing container" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018764    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018780    5424 cpu_manager.go:414] "containerMap: removing container" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018787    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018795    5424 cpu_manager.go:414] "containerMap: removing container" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018801    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018812    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018819    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018833    5424 cpu_manager.go:414] "containerMap: removing container" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018839    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018850    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018858    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018870    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018876    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018893    5424 cpu_manager.go:414] "containerMap: removing container" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018900    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018910    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018916    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018928    5424 cpu_manager.go:414] "containerMap: removing container" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018935    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018951    5424 cpu_manager.go:414] "containerMap: removing container" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018958    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018968    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018975    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018984    5424 cpu_manager.go:414] "containerMap: removing container" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.018991    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019000    5424 cpu_manager.go:414] "containerMap: removing container" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019006    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019016    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019023    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019186    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019197    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019207    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019220    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019230    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019240    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019247    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019255    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019263    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019271    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019281    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019289    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019300    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019308    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019317    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ed147e-3d13-4ec0-8b3c-c0c01098e411" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019326    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019337    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e66338-9228-4372-90ef-44c3e3b199f3" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019346    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019354    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019364    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019371    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019378    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c35c5-a969-4d15-b7d0-e965f3100439" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019387    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019395    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019403    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ad568-08e6-4811-90e3-719b385d8949" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019414    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019422    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019431    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019441    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.019949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.019449    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2deeb6c-06b5-4632-a2b8-4b4b3202d9c4" containerName="cleanup"
Nov 03 20:30:34.026228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.026201    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.044753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.044719    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.125977    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126040    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126140    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126182    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126266    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126325    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126356    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwx4\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126430    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.128593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.126457    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.228759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.228724    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.228933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.228862    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.228991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.228949    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.228996    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwx4\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229077    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229092    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229160    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229255    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229314    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229365    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229483 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229309    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229564 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229482    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.229645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.229617    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.230147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.230126    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.231734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.231711    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.231942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.231921    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.248629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.248608    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.248984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.248964    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwx4\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.347137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.347064    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:34.672196 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:34.672159    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0c8797_f1c7_4929_a88f_56c0784cd5a0.slice/crio-02eb38eb06890388e49a130c4fecf6ae230fc6c2a77777bf4627c1a13a442411 WatchSource:0}: Error finding container 02eb38eb06890388e49a130c4fecf6ae230fc6c2a77777bf4627c1a13a442411: Status 404 returned error can't find the container with id 02eb38eb06890388e49a130c4fecf6ae230fc6c2a77777bf4627c1a13a442411
Nov 03 20:30:34.674905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.674884    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:30:34.675055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.675025    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:34.675101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.675089    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:34.675141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.675116    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:34.949648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.949563    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" event={"ID":"8d0c8797-f1c7-4929-a88f-56c0784cd5a0","Type":"ContainerStarted","Data":"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"}
Nov 03 20:30:34.949648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.949626    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" event={"ID":"8d0c8797-f1c7-4929-a88f-56c0784cd5a0","Type":"ContainerStarted","Data":"02eb38eb06890388e49a130c4fecf6ae230fc6c2a77777bf4627c1a13a442411"}
Nov 03 20:30:34.994267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:34.994217    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podStartSLOduration=1.994198392 podStartE2EDuration="1.994198392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:34.99283314 +0000 UTC m=+1577.201849181" watchObservedRunningTime="2026-11-03 20:30:34.994198392 +0000 UTC m=+1577.203214462"
Nov 03 20:30:35.347984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:35.347948    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:35.349750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:35.349722    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:35.349857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:35.349792    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:36.348100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:36.348058    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:36.348648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:36.348145    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:37.215153 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:37.215097    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:37.350089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:37.350051    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:37.350806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:37.350770    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:38.348073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:38.348031    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:38.348292 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:38.348092    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:38.684406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:38.684320    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main" containerID="cri-o://a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb" gracePeriod=45
Nov 03 20:30:39.039558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.030183    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:30:39.090098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.090048    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092176    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092288    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092331    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092614    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\" (UniqueName: \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092722    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.092779    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\") pod \"85dd4a05-42d1-4b10-b649-293b1795ffa7\" (UID: \"85dd4a05-42d1-4b10-b649-293b1795ffa7\") "
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.093712    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.096572    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:39.099030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.096880    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:39.100099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.099964    5424 generic.go:356] "Generic (PLEG): container finished" podID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerID="a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb" exitCode=0
Nov 03 20:30:39.100308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.100288    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" event={"ID":"85dd4a05-42d1-4b10-b649-293b1795ffa7","Type":"ContainerDied","Data":"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"}
Nov 03 20:30:39.100403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.100332    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd" event={"ID":"85dd4a05-42d1-4b10-b649-293b1795ffa7","Type":"ContainerDied","Data":"380f3b13c080f41fc690c2d4cbfd42ef60e963659144bc27d3f65fd9f9cb1b10"}
Nov 03 20:30:39.100403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.100354    5424 scope.go:122] "RemoveContainer" containerID="a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"
Nov 03 20:30:39.102338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.102320    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"
Nov 03 20:30:39.102705 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.102679    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:39.109362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.106234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "kube-api-access-tltg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:39.109362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.107676    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm" pod "85dd4a05-42d1-4b10-b649-293b1795ffa7" (UID: "85dd4a05-42d1-4b10-b649-293b1795ffa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:39.212060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.207606    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.212461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.212379    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.212461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.212408    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tltg7\" (UniqueName: \"kubernetes.io/projected/85dd4a05-42d1-4b10-b649-293b1795ffa7-kube-api-access-tltg7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.212461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.212422    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dd4a05-42d1-4b10-b649-293b1795ffa7-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.212461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.212436    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dd4a05-42d1-4b10-b649-293b1795ffa7-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:39.212461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.211688    5424 scope.go:122] "RemoveContainer" containerID="a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"
Nov 03 20:30:39.213091 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:39.212985    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb\": container with ID starting with a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb not found: ID does not exist" containerID="a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"
Nov 03 20:30:39.213091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.213037    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb"} err="failed to get container status \"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb\": rpc error: code = NotFound desc = could not find container \"a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb\": container with ID starting with a03a1b70e1eb53439ea093e1778bfba4c505633e928b26d352818ec65b045afb not found: ID does not exist"
Nov 03 20:30:39.257992 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:39.257959    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:39.348091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.348049    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:39.348293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.348112    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:39.469300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.469268    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:30:39.481327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.481297    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-eb87f429/e2e-lifecycle-model-eb87f429-kserve-c88979d5-gshnd"]
Nov 03 20:30:39.932567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:39.930189    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" path="/var/lib/kubelet/pods/85dd4a05-42d1-4b10-b649-293b1795ffa7/volumes"
Nov 03 20:30:40.348120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:40.348080    5424 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 20:30:40.348319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:40.348152    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 20:30:41.352551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:41.352503    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:41.352979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:41.352965    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:41.353607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:41.353584    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:30:42.295162 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:42.295126    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:30:46.564297 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.564255    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"]
Nov 03 20:30:46.565006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.564990    5424 cpu_manager.go:414] "containerMap: removing container" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565008    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565023    5424 cpu_manager.go:414] "containerMap: removing container" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565028    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422fb73b-5613-4f3d-bebc-2fcbaabccc6e" containerName="cleanup"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565039    5424 cpu_manager.go:414] "containerMap: removing container" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565044    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main"
Nov 03 20:30:46.565051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565052    5424 cpu_manager.go:414] "containerMap: removing container" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565057    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565071    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565076    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565097    5424 cpu_manager.go:414] "containerMap: removing container" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565102    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab51712-e9b0-4496-a4f7-9d366975228a" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565196    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adbabc7-f309-4e4c-b02d-d7c000dff668" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565210    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aed298-5fe8-4767-ab2a-6cb3c084150e" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565216    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="617670b7-3a01-49cd-887d-4b1c6da28416" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565226    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dd4a05-42d1-4b10-b649-293b1795ffa7" containerName="main"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565231    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="547a2d0a-888e-48ce-8cb4-a7535fc422b4" containerName="cleanup"
Nov 03 20:30:46.565242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.565237    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="364e61c0-ddfc-482a-ba85-5d97a88fb413" containerName="cleanup"
Nov 03 20:30:46.570034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.570017    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.586857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.586834    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"]
Nov 03 20:30:46.592798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.592774    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:30:46.592958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.592794    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:30:46.628088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.628055    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.628573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.628546    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzclk\" (UniqueName: \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.628659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.628605    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.730656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.730614    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzclk\" (UniqueName: \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.730863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.730669    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.730863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.730738    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.731323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.731295    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.733434 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.733383    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.750310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.750283    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzclk\" (UniqueName: \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\") pod \"maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:46.885112 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:46.885028    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:47.190631 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:30:47.190589    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4787ee7a_4828_4a83_ae3c_b9f8a5317243.slice/crio-b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63 WatchSource:0}: Error finding container b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63: Status 404 returned error can't find the container with id b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63
Nov 03 20:30:47.192576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:47.192507    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"]
Nov 03 20:30:48.211569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:48.211512    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6" event={"ID":"4787ee7a-4828-4a83-ae3c-b9f8a5317243","Type":"ContainerStarted","Data":"27eac1483c175a5023d3f4c171433996698f1b1310b3624b8a77bb8c874c2627"}
Nov 03 20:30:48.212049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:48.211597    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6" event={"ID":"4787ee7a-4828-4a83-ae3c-b9f8a5317243","Type":"ContainerStarted","Data":"b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63"}
Nov 03 20:30:48.253361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:48.253313    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6" podStartSLOduration=2.253298825 podStartE2EDuration="2.253298825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:48.251097892 +0000 UTC m=+1590.460113892" watchObservedRunningTime="2026-11-03 20:30:48.253298825 +0000 UTC m=+1590.462314837"
Nov 03 20:30:49.313545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:49.312872    5424 generic.go:356] "Generic (PLEG): container finished" podID="4787ee7a-4828-4a83-ae3c-b9f8a5317243" containerID="27eac1483c175a5023d3f4c171433996698f1b1310b3624b8a77bb8c874c2627" exitCode=0
Nov 03 20:30:49.313545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:49.313138    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6" event={"ID":"4787ee7a-4828-4a83-ae3c-b9f8a5317243","Type":"ContainerDied","Data":"27eac1483c175a5023d3f4c171433996698f1b1310b3624b8a77bb8c874c2627"}
Nov 03 20:30:49.437046 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:49.437012    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:51.018110 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.017231    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:51.058855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.058821    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\") pod \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") "
Nov 03 20:30:51.059075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.058913    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\" (UniqueName: \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\") pod \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") "
Nov 03 20:30:51.059346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.059320    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\") pod \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\" (UID: \"4787ee7a-4828-4a83-ae3c-b9f8a5317243\") "
Nov 03 20:30:51.060631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.060603    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca" pod "4787ee7a-4828-4a83-ae3c-b9f8a5317243" (UID: "4787ee7a-4828-4a83-ae3c-b9f8a5317243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:51.062491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.062467    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca" pod "4787ee7a-4828-4a83-ae3c-b9f8a5317243" (UID: "4787ee7a-4828-4a83-ae3c-b9f8a5317243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:51.063202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.063181    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk" pod "4787ee7a-4828-4a83-ae3c-b9f8a5317243" (UID: "4787ee7a-4828-4a83-ae3c-b9f8a5317243"). InnerVolumeSpecName "kube-api-access-rzclk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:51.162225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.162191    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:51.162225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.162222    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4787ee7a-4828-4a83-ae3c-b9f8a5317243-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:51.162225 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.162232    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzclk\" (UniqueName: \"kubernetes.io/projected/4787ee7a-4828-4a83-ae3c-b9f8a5317243-kube-api-access-rzclk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:30:51.385419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.383284    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6" event={"ID":"4787ee7a-4828-4a83-ae3c-b9f8a5317243","Type":"ContainerDied","Data":"b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63"}
Nov 03 20:30:51.385419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.383378    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69848cddb88f72108d0613cab2557c2dba78466859727fc44aa6bb925a58b63"
Nov 03 20:30:51.385419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:51.383492    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"
Nov 03 20:30:52.139661 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:52.137140    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:53.330028 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:53.329978    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:53.345163 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:53.345113    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:30:54.131237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:54.131204    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"]
Nov 03 20:30:54.176365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:54.176306    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4cdbeb-5878l"]
Nov 03 20:30:55.933230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:30:55.933195    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50145b6f-5d18-49b6-8cb7-a3b47c0c7356" path="/var/lib/kubelet/pods/50145b6f-5d18-49b6-8cb7-a3b47c0c7356/volumes"
Nov 03 20:30:59.521524 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:30:59.521489    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:31:00.829424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:00.829387    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:31:00.830050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:00.829602    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" containerID="cri-o://89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e" gracePeriod=30
Nov 03 20:31:01.264096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.264053    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:31:01.264301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.264254    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" containerID="cri-o://f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b" gracePeriod=30
Nov 03 20:31:01.362287 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.362255    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:31:01.363139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.363095    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" containerID="cri-o://3b945ab47bce72de8f2689b8fbae96005359fb8000fc8431615c0807b767ee6b" gracePeriod=30
Nov 03 20:31:01.406210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.406141    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:31:01.406468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:01.406426    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" containerID="cri-o://2bde411639307235606954eb8d13aadbaacff4c11f1725585223580b1960a945" gracePeriod=30
Nov 03 20:31:02.876299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:02.876262    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:02.876936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:02.876909    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:03.154318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.154225    5424 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:03.154318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.154287    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:03.302899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.293938    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:31:03.302899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.294181    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" containerID="cri-o://ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b" gracePeriod=30
Nov 03 20:31:03.964113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.964076    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:03.964492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:03.964147    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:05.041556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:05.036191    5424 patch_prober.go:29] interesting pod/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:05.041556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:05.036251    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:05.142714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:05.142286    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-85qct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:05.142714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:05.142347    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:06.211616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.208723    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:31:06.315052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315020    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315084    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315127    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315157    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315218    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315547 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315495    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315631 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315601    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315689 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315671    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.315743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.315714    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\") pod \"18de12c9-5db1-42b4-9f23-a18932e57d95\" (UID: \"18de12c9-5db1-42b4-9f23-a18932e57d95\") "
Nov 03 20:31:06.316940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.316906    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.317306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.317246    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:06.317569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.317507    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.317800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.317776    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.318304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.318267    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.319178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.319152    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.319351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.319309    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:06.320592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.320567    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "kube-api-access-9mq25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.322092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.322068    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy" pod "18de12c9-5db1-42b4-9f23-a18932e57d95" (UID: "18de12c9-5db1-42b4-9f23-a18932e57d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.418815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418777    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418819    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418832    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18de12c9-5db1-42b4-9f23-a18932e57d95-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418843    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418855    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418867    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418877    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418889    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18de12c9-5db1-42b4-9f23-a18932e57d95-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.418966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.418899    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mq25\" (UniqueName: \"kubernetes.io/projected/18de12c9-5db1-42b4-9f23-a18932e57d95-kube-api-access-9mq25\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.579951 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.579919    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:31:06.659813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.659717    5424 generic.go:356] "Generic (PLEG): container finished" podID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerID="89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e" exitCode=0
Nov 03 20:31:06.659983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.659847    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" event={"ID":"18de12c9-5db1-42b4-9f23-a18932e57d95","Type":"ContainerDied","Data":"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"}
Nov 03 20:31:06.659983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.659888    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2" event={"ID":"18de12c9-5db1-42b4-9f23-a18932e57d95","Type":"ContainerDied","Data":"3f5138b81be63cd76c1c8a9ef5a3b8e39ad37aa0d481cd012e6c7b0fb63d7444"}
Nov 03 20:31:06.659983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.659914    5424 scope.go:122] "RemoveContainer" containerID="89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"
Nov 03 20:31:06.660153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.660049    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"
Nov 03 20:31:06.671707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.671677    5424 generic.go:356] "Generic (PLEG): container finished" podID="408b216c-7209-4321-9215-86d5cc0c6455" containerID="2bde411639307235606954eb8d13aadbaacff4c11f1725585223580b1960a945" exitCode=0
Nov 03 20:31:06.671914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.671879    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" event={"ID":"408b216c-7209-4321-9215-86d5cc0c6455","Type":"ContainerDied","Data":"2bde411639307235606954eb8d13aadbaacff4c11f1725585223580b1960a945"}
Nov 03 20:31:06.674248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.674219    5424 generic.go:356] "Generic (PLEG): container finished" podID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerID="f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b" exitCode=0
Nov 03 20:31:06.674521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.674501    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"
Nov 03 20:31:06.674748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.674723    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" event={"ID":"004f5ef7-6b47-4b5e-ab76-69d20d132603","Type":"ContainerDied","Data":"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"}
Nov 03 20:31:06.674840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.674775    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp" event={"ID":"004f5ef7-6b47-4b5e-ab76-69d20d132603","Type":"ContainerDied","Data":"6621093b88325c4618c780ed6bba28b2c8051d99585731db70ac63a62b72ba33"}
Nov 03 20:31:06.677626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.677599    5424 generic.go:356] "Generic (PLEG): container finished" podID="437fef2f-8f3c-4208-8a72-54807de6816e" containerID="3b945ab47bce72de8f2689b8fbae96005359fb8000fc8431615c0807b767ee6b" exitCode=0
Nov 03 20:31:06.677742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.677656    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" event={"ID":"437fef2f-8f3c-4208-8a72-54807de6816e","Type":"ContainerDied","Data":"3b945ab47bce72de8f2689b8fbae96005359fb8000fc8431615c0807b767ee6b"}
Nov 03 20:31:06.695780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.695019    5424 scope.go:122] "RemoveContainer" containerID="89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"
Nov 03 20:31:06.696031 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:06.696000    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e\": container with ID starting with 89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e not found: ID does not exist" containerID="89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"
Nov 03 20:31:06.696135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.696041    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e"} err="failed to get container status \"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e\": rpc error: code = NotFound desc = could not find container \"89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e\": container with ID starting with 89249a57b5a1e8843033ff04e1fce6223e0aa1912e4f52c9160a34cba39a2a7e not found: ID does not exist"
Nov 03 20:31:06.696135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.696066    5424 scope.go:122] "RemoveContainer" containerID="f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"
Nov 03 20:31:06.723800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.723774    5424 scope.go:122] "RemoveContainer" containerID="f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"
Nov 03 20:31:06.728227 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:06.727774    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b\": container with ID starting with f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b not found: ID does not exist" containerID="f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"
Nov 03 20:31:06.728364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.728247    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b"} err="failed to get container status \"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b\": rpc error: code = NotFound desc = could not find container \"f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b\": container with ID starting with f16158b7d21df783eac8ad60f17c33caacf96861806cf904edbdf5ad7a240b2b not found: ID does not exist"
Nov 03 20:31:06.738556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.738508    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739382    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739465    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739507    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739630    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739722    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739784    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739808    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.739861    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.741346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.740020    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\") pod \"004f5ef7-6b47-4b5e-ab76-69d20d132603\" (UID: \"004f5ef7-6b47-4b5e-ab76-69d20d132603\") "
Nov 03 20:31:06.743736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.743425    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.743736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.743628    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:06.743947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.743923    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.744102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.744060    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.744213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.744132    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.753842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.753814    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:31:06.758075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.758004    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.758075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.758038    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "kube-api-access-tssz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.761802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.761757    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-nmfb2"]
Nov 03 20:31:06.762156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.762127    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.762761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.762706    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo" pod "004f5ef7-6b47-4b5e-ab76-69d20d132603" (UID: "004f5ef7-6b47-4b5e-ab76-69d20d132603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:06.846553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.846187    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "kube-api-access-htm8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.848130 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848103    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848175    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848304    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848368    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848455    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848490    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848544    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.848744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848663    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.849142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848899    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\") pod \"437fef2f-8f3c-4208-8a72-54807de6816e\" (UID: \"437fef2f-8f3c-4208-8a72-54807de6816e\") "
Nov 03 20:31:06.849142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.848966    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:06.849142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849096    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.849142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849110    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.849421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849238    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.849421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849355    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.849807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849785    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004f5ef7-6b47-4b5e-ab76-69d20d132603-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849810    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-kube-api-access-tssz9\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849824    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849838    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849852    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849866    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.849890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849881    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849893    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849907    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htm8p\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-kube-api-access-htm8p\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849921    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849934    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849946    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004f5ef7-6b47-4b5e-ab76-69d20d132603-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849960    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/437fef2f-8f3c-4208-8a72-54807de6816e-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849973    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.850213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.849986    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.851155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.851127    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:06.852806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.852780    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.852944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.852926    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy" pod "437fef2f-8f3c-4208-8a72-54807de6816e" (UID: "437fef2f-8f3c-4208-8a72-54807de6816e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.925485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.925411    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:31:06.951769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.951734    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/437fef2f-8f3c-4208-8a72-54807de6816e-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.951769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.951764    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/437fef2f-8f3c-4208-8a72-54807de6816e-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:06.951769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:06.951773    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/437fef2f-8f3c-4208-8a72-54807de6816e-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.028958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.028902    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:31:07.036439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.036399    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-zvqpp"]
Nov 03 20:31:07.052930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.052899    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.052987    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053058    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053115    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053163    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053237    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053244    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:07.053314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053265    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053262    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:07.053659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053354    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053389    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\") pod \"408b216c-7209-4321-9215-86d5cc0c6455\" (UID: \"408b216c-7209-4321-9215-86d5cc0c6455\") "
Nov 03 20:31:07.053659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053520    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:07.053839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053638    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:07.053839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.053710    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:07.054827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.054794    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.054827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.054826    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408b216c-7209-4321-9215-86d5cc0c6455-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.055017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.054840    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.055017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.054854    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.055017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.054868    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.056032 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.056007    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:07.056116 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.056063    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "kube-api-access-9zvwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:07.056234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.056214    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:07.056295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.056243    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token" pod "408b216c-7209-4321-9215-86d5cc0c6455" (UID: "408b216c-7209-4321-9215-86d5cc0c6455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:07.156187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.156149    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408b216c-7209-4321-9215-86d5cc0c6455-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.156187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.156179    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvwh\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-kube-api-access-9zvwh\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.156187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.156190    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408b216c-7209-4321-9215-86d5cc0c6455-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.156187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.156199    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408b216c-7209-4321-9215-86d5cc0c6455-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:07.179884 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:07.179790    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:31:07.688312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.688270    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm" event={"ID":"437fef2f-8f3c-4208-8a72-54807de6816e","Type":"ContainerDied","Data":"d6150983f9bf3135a72111b86380d2e9ed5f4242367e422b17dc3a20307e43c8"}
Nov 03 20:31:07.688950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.688307    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"
Nov 03 20:31:07.689103 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.688965    5424 scope.go:122] "RemoveContainer" containerID="3b945ab47bce72de8f2689b8fbae96005359fb8000fc8431615c0807b767ee6b"
Nov 03 20:31:07.691723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.691695    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct" event={"ID":"408b216c-7209-4321-9215-86d5cc0c6455","Type":"ContainerDied","Data":"4e0ee0a9bc62b0f1f9f5cf9a45dbb4c2b4ad906a2b661049cb721c8f8f8d12c7"}
Nov 03 20:31:07.691854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.691840    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"
Nov 03 20:31:07.704763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.703923    5424 scope.go:122] "RemoveContainer" containerID="2bde411639307235606954eb8d13aadbaacff4c11f1725585223580b1960a945"
Nov 03 20:31:07.759292 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.759255    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:31:07.772019 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.771988    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-85qct"]
Nov 03 20:31:07.807120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.807079    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:31:07.811544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.811503    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-svjkm"]
Nov 03 20:31:07.922589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.922559    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" path="/var/lib/kubelet/pods/004f5ef7-6b47-4b5e-ab76-69d20d132603/volumes"
Nov 03 20:31:07.923145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.923131    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" path="/var/lib/kubelet/pods/18de12c9-5db1-42b4-9f23-a18932e57d95/volumes"
Nov 03 20:31:07.923638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.923623    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408b216c-7209-4321-9215-86d5cc0c6455" path="/var/lib/kubelet/pods/408b216c-7209-4321-9215-86d5cc0c6455/volumes"
Nov 03 20:31:07.924406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:07.924390    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" path="/var/lib/kubelet/pods/437fef2f-8f3c-4208-8a72-54807de6816e/volumes"
Nov 03 20:31:08.597569 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.597498    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:31:08.704875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.704842    5424 generic.go:356] "Generic (PLEG): container finished" podID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerID="ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b" exitCode=0
Nov 03 20:31:08.705335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.704910    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"
Nov 03 20:31:08.705335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.705031    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerDied","Data":"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"}
Nov 03 20:31:08.705335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.705082    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp" event={"ID":"0ce3a833-2953-46fa-a2f5-19bc942e3cf9","Type":"ContainerDied","Data":"eaf842b06ff26bd05340eadc44eec2cd90d6a614f39fcba80c08466506979079"}
Nov 03 20:31:08.705335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.705116    5424 scope.go:122] "RemoveContainer" containerID="ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"
Nov 03 20:31:08.721518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.721492    5424 scope.go:122] "RemoveContainer" containerID="d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"
Nov 03 20:31:08.735360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.735338    5424 scope.go:122] "RemoveContainer" containerID="ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"
Nov 03 20:31:08.735792 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:08.735761    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b\": container with ID starting with ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b not found: ID does not exist" containerID="ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"
Nov 03 20:31:08.735908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.735804    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b"} err="failed to get container status \"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b\": rpc error: code = NotFound desc = could not find container \"ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b\": container with ID starting with ebc29c0af609bc8e976ff7ec3ebe365416e9b6769312d16794f3b6514d3af08b not found: ID does not exist"
Nov 03 20:31:08.735908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.735829    5424 scope.go:122] "RemoveContainer" containerID="d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"
Nov 03 20:31:08.736211 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:08.736183    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae\": container with ID starting with d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae not found: ID does not exist" containerID="d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"
Nov 03 20:31:08.736295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.736227    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae"} err="failed to get container status \"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae\": rpc error: code = NotFound desc = could not find container \"d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae\": container with ID starting with d6fa5b8ad5ce8c51c9e02a4e7b2e46e170ccae804ad9c04fb34bd54bf38741ae not found: ID does not exist"
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787542    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787615    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787674    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787746    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787798    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787872    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.787920    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.788022    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.788176    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\") pod \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\" (UID: \"0ce3a833-2953-46fa-a2f5-19bc942e3cf9\") "
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.797821    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.798171    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.798996    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.799057    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:08.824604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.799105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:08.842556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.825788    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:08.843016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.836725    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:08.843102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.836801    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "kube-api-access-rdvll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:08.843167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.838044    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy" pod "0ce3a833-2953-46fa-a2f5-19bc942e3cf9" (UID: "0ce3a833-2953-46fa-a2f5-19bc942e3cf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899651    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvll\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-kube-api-access-rdvll\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899690    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899706    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899719    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899733    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899746    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899760    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899772    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:08.906551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:08.899785    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3a833-2953-46fa-a2f5-19bc942e3cf9-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:09.078920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:09.078836    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:31:09.088426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:09.088397    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4cdbeb-openshift-default-6cb77c8559-6xrsp"]
Nov 03 20:31:09.536835 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:09.536794    5424 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-conmon-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbabc7_f309_4e4c_b02d_d7c000dff668.slice/crio-e3aea903ac2e783b98fff63e4fa99e8caf2ca6fcfe13fa1229b1f8472312fe9d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:31:09.922070 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:09.921989    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" path="/var/lib/kubelet/pods/0ce3a833-2953-46fa-a2f5-19bc942e3cf9/volumes"
Nov 03 20:31:11.248308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.248266    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:31:11.248851 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.248631    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy" containerID="cri-o://434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597" gracePeriod=30
Nov 03 20:31:11.260415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.260381    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:31:11.261568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.261478    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" containerID="cri-o://a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759" gracePeriod=30
Nov 03 20:31:11.267433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.267404    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:31:11.267735 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:11.267705    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" containerID="cri-o://983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb" gracePeriod=30
Nov 03 20:31:12.287935 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:12.287893    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:31:13.163764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:13.163724    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:13.163964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:13.163803    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:15.032475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.032438    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"]
Nov 03 20:31:15.033369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033346    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4787ee7a-4828-4a83-ae3c-b9f8a5317243" containerName="revoke-keys"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033370    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4787ee7a-4828-4a83-ae3c-b9f8a5317243" containerName="revoke-keys"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033429    5424 cpu_manager.go:414] "containerMap: removing container" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033438    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033459    5424 cpu_manager.go:414] "containerMap: removing container" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033466    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033484    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033491    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033506    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033514    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033550    5424 cpu_manager.go:414] "containerMap: removing container" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033558    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033569    5424 cpu_manager.go:414] "containerMap: removing container" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033575    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033759    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033777    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce3a833-2953-46fa-a2f5-19bc942e3cf9" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033788    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="004f5ef7-6b47-4b5e-ab76-69d20d132603" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033798    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="408b216c-7209-4321-9215-86d5cc0c6455" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033811    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4787ee7a-4828-4a83-ae3c-b9f8a5317243" containerName="revoke-keys"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033827    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="437fef2f-8f3c-4208-8a72-54807de6816e" containerName="istio-proxy"
Nov 03 20:31:15.033904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.033838    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="18de12c9-5db1-42b4-9f23-a18932e57d95" containerName="istio-proxy"
Nov 03 20:31:15.039073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.039050    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.100510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.100470    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"]
Nov 03 20:31:15.113484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.113444    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.113701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.113638    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq499\" (UniqueName: \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.113937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.113913    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.126810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.126769    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:15.126810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.126792    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:31:15.215777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.215739    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.216028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.215894    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.216028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.215927    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq499\" (UniqueName: \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.216761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.216734    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.218301 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.218282    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.245235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.245204    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq499\" (UniqueName: \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\") pod \"maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.272105 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.272067    5424 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:15.272238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.272128    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:15.354357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.354272    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:15.676375 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:31:15.676339    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705f3fbf_1914_4aa4_97e1_60a4635974cb.slice/crio-d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6 WatchSource:0}: Error finding container d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6: Status 404 returned error can't find the container with id d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6
Nov 03 20:31:15.677521 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.677431    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"]
Nov 03 20:31:15.805561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:15.805419    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d" event={"ID":"705f3fbf-1914-4aa4-97e1-60a4635974cb","Type":"ContainerStarted","Data":"d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6"}
Nov 03 20:31:16.596922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.596892    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:31:16.651493 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651458    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.651699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651540    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.651699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651654    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.651927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651758    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.651927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651798    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.651927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.651852    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.652078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.652007    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.652136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.652091    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.652198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.652165    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\") pod \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\" (UID: \"86f73b3c-a68a-4433-9ee0-e5f67f7edff6\") "
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.655619    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.655838    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.655917    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.656010    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.656105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.658418 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.656088    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.659723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.659698    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.660156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.660132    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.663011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.662949    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x" pod "86f73b3c-a68a-4433-9ee0-e5f67f7edff6" (UID: "86f73b3c-a68a-4433-9ee0-e5f67f7edff6"). InnerVolumeSpecName "kube-api-access-9mq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.713580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.713553    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:31:16.722184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.722158    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755557    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755630    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755684    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755789    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755835    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.755866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.755876    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.756344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756003    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.756344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756031    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.756344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756063    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756724    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756841    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756908    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.756984    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757053    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757085    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757149    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757194    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757273    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\") pod \"e072a74d-12c7-4927-abfc-bd476d09763e\" (UID: \"e072a74d-12c7-4927-abfc-bd476d09763e\") "
Nov 03 20:31:16.761191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.757345    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\") pod \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\" (UID: \"4fc8264e-3e1b-452b-ad1d-fe7527c85cac\") "
Nov 03 20:31:16.762097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762018    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762234    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762337    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762430    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762519    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762708    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mq7x\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-kube-api-access-9mq7x\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.762830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.762816    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.793867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.793831    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.798129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.798089    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.798337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.798324    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f73b3c-a68a-4433-9ee0-e5f67f7edff6-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.798451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.777254    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.798715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.780020    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.798818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.780323    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.798936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.780610    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.799034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.782616    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.799150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.791638    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.799264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.795224    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.799376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.795901    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.799480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.796408    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:16.799683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.797757    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "kube-api-access-nblnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.804402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.804361    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:16.804736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.804688    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.804999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.804962    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:16.805316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.805280    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "kube-api-access-wxcdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.811356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.809461    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.811356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.809593    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token" pod "e072a74d-12c7-4927-abfc-bd476d09763e" (UID: "e072a74d-12c7-4927-abfc-bd476d09763e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.811356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.811086    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token" pod "4fc8264e-3e1b-452b-ad1d-fe7527c85cac" (UID: "4fc8264e-3e1b-452b-ad1d-fe7527c85cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.822819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.822188    5424 generic.go:356] "Generic (PLEG): container finished" podID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerID="434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597" exitCode=0
Nov 03 20:31:16.822819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.822470    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" event={"ID":"4fc8264e-3e1b-452b-ad1d-fe7527c85cac","Type":"ContainerDied","Data":"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"}
Nov 03 20:31:16.822819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.822588    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp" event={"ID":"4fc8264e-3e1b-452b-ad1d-fe7527c85cac","Type":"ContainerDied","Data":"cd445ec295db5c1eaa20ea86cc5f20a0a113ce789ff438298766a317a0e0c56f"}
Nov 03 20:31:16.822819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.822625    5424 scope.go:122] "RemoveContainer" containerID="434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"
Nov 03 20:31:16.822819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.822722    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"
Nov 03 20:31:16.830753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.830434    5424 generic.go:356] "Generic (PLEG): container finished" podID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerID="a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759" exitCode=0
Nov 03 20:31:16.831545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.830620    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" event={"ID":"86f73b3c-a68a-4433-9ee0-e5f67f7edff6","Type":"ContainerDied","Data":"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"}
Nov 03 20:31:16.831545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.831059    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb" event={"ID":"86f73b3c-a68a-4433-9ee0-e5f67f7edff6","Type":"ContainerDied","Data":"e9370851670c12f351ecca029c5fadbb971a14e9704cb47fa4243af138b14b9d"}
Nov 03 20:31:16.831545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.831172    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"
Nov 03 20:31:16.845316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.843478    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"
Nov 03 20:31:16.849318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.849218    5424 scope.go:122] "RemoveContainer" containerID="434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"
Nov 03 20:31:16.849972 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:16.849783    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597\": container with ID starting with 434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597 not found: ID does not exist" containerID="434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"
Nov 03 20:31:16.849972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.849820    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597"} err="failed to get container status \"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597\": rpc error: code = NotFound desc = could not find container \"434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597\": container with ID starting with 434c75a7ab2737ab85c4bc9c20ed84fd952876f73880763d062e5c5baa7d4597 not found: ID does not exist"
Nov 03 20:31:16.849972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.849844    5424 scope.go:122] "RemoveContainer" containerID="a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"
Nov 03 20:31:16.851100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.850926    5424 generic.go:356] "Generic (PLEG): container finished" podID="e072a74d-12c7-4927-abfc-bd476d09763e" containerID="983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb" exitCode=0
Nov 03 20:31:16.851100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.851057    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" event={"ID":"e072a74d-12c7-4927-abfc-bd476d09763e","Type":"ContainerDied","Data":"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"}
Nov 03 20:31:16.851262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.851111    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7" event={"ID":"e072a74d-12c7-4927-abfc-bd476d09763e","Type":"ContainerDied","Data":"e775a3a347c8e4d900c8889c178b09276713faa166e411f91fa39e52938f204f"}
Nov 03 20:31:16.856640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.855400    5424 generic.go:356] "Generic (PLEG): container finished" podID="705f3fbf-1914-4aa4-97e1-60a4635974cb" containerID="0465536022db2939cc8e0764c976320f0ab074881d6a1f297730a2177606d31e" exitCode=0
Nov 03 20:31:16.856640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.855602    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d" event={"ID":"705f3fbf-1914-4aa4-97e1-60a4635974cb","Type":"ContainerDied","Data":"0465536022db2939cc8e0764c976320f0ab074881d6a1f297730a2177606d31e"}
Nov 03 20:31:16.880221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.880192    5424 scope.go:122] "RemoveContainer" containerID="a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"
Nov 03 20:31:16.880677 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:16.880650    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759\": container with ID starting with a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759 not found: ID does not exist" containerID="a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"
Nov 03 20:31:16.880840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.880689    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759"} err="failed to get container status \"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759\": rpc error: code = NotFound desc = could not find container \"a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759\": container with ID starting with a944b4921f0ccd0fa75164968315a55565d162b3023c367356163cab23b46759 not found: ID does not exist"
Nov 03 20:31:16.880840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.880710    5424 scope.go:122] "RemoveContainer" containerID="983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"
Nov 03 20:31:16.894370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.894340    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:31:16.897030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.897012    5424 scope.go:122] "RemoveContainer" containerID="983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"
Nov 03 20:31:16.897376 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:16.897350    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb\": container with ID starting with 983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb not found: ID does not exist" containerID="983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"
Nov 03 20:31:16.897494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.897389    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb"} err="failed to get container status \"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb\": rpc error: code = NotFound desc = could not find container \"983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb\": container with ID starting with 983b6af31eee53c4838307a1e7d10a2af2af6113d62ffd893ad4079d686a7fdb not found: ID does not exist"
Nov 03 20:31:16.899796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.899774    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-9hxjp"]
Nov 03 20:31:16.900478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900455    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900481    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900491    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900499    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900508    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e072a74d-12c7-4927-abfc-bd476d09763e-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900517    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900566    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900580    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900593    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900604    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900613    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900622    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900630    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e072a74d-12c7-4927-abfc-bd476d09763e-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900641    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcdp\" (UniqueName: \"kubernetes.io/projected/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-kube-api-access-wxcdp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900649    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e072a74d-12c7-4927-abfc-bd476d09763e-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900658    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nblnc\" (UniqueName: \"kubernetes.io/projected/e072a74d-12c7-4927-abfc-bd476d09763e-kube-api-access-nblnc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.900831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.900666    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc8264e-3e1b-452b-ad1d-fe7527c85cac-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.958669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.958630    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:31:16.969618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:16.969563    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-8tggb"]
Nov 03 20:31:17.092869 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:17.092770    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:31:17.093129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:17.093113    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zmfw7"]
Nov 03 20:31:17.923084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:17.923052    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" path="/var/lib/kubelet/pods/4fc8264e-3e1b-452b-ad1d-fe7527c85cac/volumes"
Nov 03 20:31:17.923574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:17.923513    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" path="/var/lib/kubelet/pods/86f73b3c-a68a-4433-9ee0-e5f67f7edff6/volumes"
Nov 03 20:31:17.931726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:17.929980    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" path="/var/lib/kubelet/pods/e072a74d-12c7-4927-abfc-bd476d09763e/volumes"
Nov 03 20:31:18.061114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.061086    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:18.136350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.136258    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\") pod \"705f3fbf-1914-4aa4-97e1-60a4635974cb\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") "
Nov 03 20:31:18.136593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.136418    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\") pod \"705f3fbf-1914-4aa4-97e1-60a4635974cb\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") "
Nov 03 20:31:18.136707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.136628    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\" (UniqueName: \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\") pod \"705f3fbf-1914-4aa4-97e1-60a4635974cb\" (UID: \"705f3fbf-1914-4aa4-97e1-60a4635974cb\") "
Nov 03 20:31:18.136850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.136820    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca" pod "705f3fbf-1914-4aa4-97e1-60a4635974cb" (UID: "705f3fbf-1914-4aa4-97e1-60a4635974cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:18.137334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.137309    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:18.139030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.138996    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca" pod "705f3fbf-1914-4aa4-97e1-60a4635974cb" (UID: "705f3fbf-1914-4aa4-97e1-60a4635974cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:18.139030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.139009    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499" pod "705f3fbf-1914-4aa4-97e1-60a4635974cb" (UID: "705f3fbf-1914-4aa4-97e1-60a4635974cb"). InnerVolumeSpecName "kube-api-access-sq499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:18.239133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.239101    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705f3fbf-1914-4aa4-97e1-60a4635974cb-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:18.239133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.239128    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq499\" (UniqueName: \"kubernetes.io/projected/705f3fbf-1914-4aa4-97e1-60a4635974cb-kube-api-access-sq499\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:18.609910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.609875    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:31:18.610122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.610076    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" containerID="cri-o://2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b" gracePeriod=30
Nov 03 20:31:18.880511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.880430    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"
Nov 03 20:31:18.880731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.880428    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d" event={"ID":"705f3fbf-1914-4aa4-97e1-60a4635974cb","Type":"ContainerDied","Data":"d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6"}
Nov 03 20:31:18.880731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:18.880610    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cf29e74340193a06bb12c3c83a82f8dbebeb78f13e257d0a975ee2e35a0cf6"
Nov 03 20:31:19.527133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:19.527088    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:19.527509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:19.527150    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:21.023817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:21.023784    5424 scope.go:122] "RemoveContainer" containerID="5638c17d84dc28791192fd9a249d6d3e3cd9398a67a8fb85b389d4071de4f377"
Nov 03 20:31:23.897492 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.897464    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:31:23.923278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.923238    5424 generic.go:356] "Generic (PLEG): container finished" podID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerID="2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b" exitCode=0
Nov 03 20:31:23.923447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.923325    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"
Nov 03 20:31:23.923853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.923830    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" event={"ID":"603a50c1-8df7-41c6-8e8f-df4717091b34","Type":"ContainerDied","Data":"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"}
Nov 03 20:31:23.923943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.923883    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5" event={"ID":"603a50c1-8df7-41c6-8e8f-df4717091b34","Type":"ContainerDied","Data":"4800abf507115a81be232a2a24ecfe97c5df19fc0f59f2ef37956690065d3484"}
Nov 03 20:31:23.923943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.923909    5424 scope.go:122] "RemoveContainer" containerID="2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"
Nov 03 20:31:23.931629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931606    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.931783 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931769    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.931828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931794    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.931861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931841    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:23.931922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931907    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.931961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931942    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.932007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.931986    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.932065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932042    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.932065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932049    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:23.932173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932086    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.932173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932133    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\") pod \"603a50c1-8df7-41c6-8e8f-df4717091b34\" (UID: \"603a50c1-8df7-41c6-8e8f-df4717091b34\") "
Nov 03 20:31:23.932280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932223    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:23.932628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932594    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:23.932835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.932776    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:23.934360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934093    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:23.934360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934119    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:23.934360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934131    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603a50c1-8df7-41c6-8e8f-df4717091b34-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:23.934360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934142    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:23.934360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934154    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:23.934698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934576    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:23.934698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934575    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:23.934698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.934605    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "kube-api-access-lj6rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:23.936859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.936834    5424 scope.go:122] "RemoveContainer" containerID="2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"
Nov 03 20:31:23.937174 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:23.937156    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b\": container with ID starting with 2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b not found: ID does not exist" containerID="2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"
Nov 03 20:31:23.937252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.937183    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b"} err="failed to get container status \"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b\": rpc error: code = NotFound desc = could not find container \"2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b\": container with ID starting with 2e1abc1322d0458a00b4a9265330b4e578120f81c7d966ceb7d736d058a3333b not found: ID does not exist"
Nov 03 20:31:23.937323 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:23.937305    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo" pod "603a50c1-8df7-41c6-8e8f-df4717091b34" (UID: "603a50c1-8df7-41c6-8e8f-df4717091b34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:24.035621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.035582    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj6rp\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-kube-api-access-lj6rp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:24.035621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.035614    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:24.035621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.035629    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:24.035873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.035643    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603a50c1-8df7-41c6-8e8f-df4717091b34-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:24.271546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.271507    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:31:24.283819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:24.283786    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-hwdr5"]
Nov 03 20:31:25.922127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:25.922088    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" path="/var/lib/kubelet/pods/603a50c1-8df7-41c6-8e8f-df4717091b34/volumes"
Nov 03 20:31:42.286519 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:42.286433    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:31:43.292554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.290624    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:31:43.292554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.290881    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" containerID="cri-o://5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669" gracePeriod=30
Nov 03 20:31:43.303450 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.303417    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:31:43.303771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.303726    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy" containerID="cri-o://0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f" gracePeriod=30
Nov 03 20:31:43.316735 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.316694    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:31:43.317009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.316962    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy" containerID="cri-o://b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c" gracePeriod=30
Nov 03 20:31:43.326337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.326301    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:31:43.326614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.326585    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy" containerID="cri-o://84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc" gracePeriod=30
Nov 03 20:31:43.391980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.391946    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:43.392168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.392006    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:43.548945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.548864    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"]
Nov 03 20:31:43.549479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549467    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy"
Nov 03 20:31:43.549572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549481    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy"
Nov 03 20:31:43.549572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549515    5424 cpu_manager.go:414] "containerMap: removing container" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy"
Nov 03 20:31:43.549572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549522    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy"
Nov 03 20:31:43.549572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549549    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy"
Nov 03 20:31:43.549572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549554    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549589    5424 cpu_manager.go:414] "containerMap: removing container" podUID="705f3fbf-1914-4aa4-97e1-60a4635974cb" containerName="revoke-keys"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549594    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705f3fbf-1914-4aa4-97e1-60a4635974cb" containerName="revoke-keys"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549605    5424 cpu_manager.go:414] "containerMap: removing container" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549610    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549695    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="705f3fbf-1914-4aa4-97e1-60a4635974cb" containerName="revoke-keys"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549706    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="603a50c1-8df7-41c6-8e8f-df4717091b34" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549721    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e072a74d-12c7-4927-abfc-bd476d09763e" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549733    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f73b3c-a68a-4433-9ee0-e5f67f7edff6" containerName="istio-proxy"
Nov 03 20:31:43.550120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.549740    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc8264e-3e1b-452b-ad1d-fe7527c85cac" containerName="istio-proxy"
Nov 03 20:31:43.553345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.553329    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.571977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.571949    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:43.572149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.572069    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.572149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.572113    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:31:43.572350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.572333    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.572419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.572372    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4zc\" (UniqueName: \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.573642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.573616    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"]
Nov 03 20:31:43.673786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.673745    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.673973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.673797    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4zc\" (UniqueName: \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.673973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.673896    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.674405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.674382    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.676513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.676491    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.693843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.693808    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4zc\" (UniqueName: \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\") pod \"maas-api-revoke-sub-f98f9ea2-jhnhh\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:43.868406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:43.868314    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:44.177261 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:31:44.177227    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6240bd_4a29_4986_b8d4_b6158da09a5c.slice/crio-f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113 WatchSource:0}: Error finding container f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113: Status 404 returned error can't find the container with id f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113
Nov 03 20:31:44.178967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:44.178872    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"]
Nov 03 20:31:44.191555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:44.191510    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh" event={"ID":"5d6240bd-4a29-4986-b8d4-b6158da09a5c","Type":"ContainerStarted","Data":"f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113"}
Nov 03 20:31:45.201388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:45.201325    5424 generic.go:356] "Generic (PLEG): container finished" podID="5d6240bd-4a29-4986-b8d4-b6158da09a5c" containerID="30e4a7d3713c07ad69a3513c446d6eea486c2f622058edf1222a71d96c8d908c" exitCode=0
Nov 03 20:31:45.201899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:45.201390    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh" event={"ID":"5d6240bd-4a29-4986-b8d4-b6158da09a5c","Type":"ContainerDied","Data":"30e4a7d3713c07ad69a3513c446d6eea486c2f622058edf1222a71d96c8d908c"}
Nov 03 20:31:46.441227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.441198    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:46.694777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.692142    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\") pod \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") "
Nov 03 20:31:46.694777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.692270    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\" (UniqueName: \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\") pod \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") "
Nov 03 20:31:46.694777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.692327    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\") pod \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\" (UID: \"5d6240bd-4a29-4986-b8d4-b6158da09a5c\") "
Nov 03 20:31:46.706555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.703085    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca" pod "5d6240bd-4a29-4986-b8d4-b6158da09a5c" (UID: "5d6240bd-4a29-4986-b8d4-b6158da09a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:46.723593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.713300    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca" pod "5d6240bd-4a29-4986-b8d4-b6158da09a5c" (UID: "5d6240bd-4a29-4986-b8d4-b6158da09a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:46.893245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.888759    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:46.893245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.888798    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d6240bd-4a29-4986-b8d4-b6158da09a5c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:46.894175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.894105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc" pod "5d6240bd-4a29-4986-b8d4-b6158da09a5c" (UID: "5d6240bd-4a29-4986-b8d4-b6158da09a5c"). InnerVolumeSpecName "kube-api-access-9l4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:46.950884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.950802    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"]
Nov 03 20:31:46.957206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.957168    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-addf1a-fqjq6"]
Nov 03 20:31:46.990355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:46.990274    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4zc\" (UniqueName: \"kubernetes.io/projected/5d6240bd-4a29-4986-b8d4-b6158da09a5c-kube-api-access-9l4zc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:47.219166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:47.219085    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"
Nov 03 20:31:47.219166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:47.219128    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh" event={"ID":"5d6240bd-4a29-4986-b8d4-b6158da09a5c","Type":"ContainerDied","Data":"f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113"}
Nov 03 20:31:47.219392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:47.219177    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3a7b418901f9cb333f9876a58445a76ef72f7eebfc81df61081a1d4390f8113"
Nov 03 20:31:47.920482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:47.920436    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4787ee7a-4828-4a83-ae3c-b9f8a5317243" path="/var/lib/kubelet/pods/4787ee7a-4828-4a83-ae3c-b9f8a5317243/volumes"
Nov 03 20:31:48.713166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.713135    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:31:48.823185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823151    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.823362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823199    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.823362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823244    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.823581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823555    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.823742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823282    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.823867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823842    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.823934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823897    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.823991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.823934    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.824131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824099    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.824200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824144    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.824326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824307    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.824395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824348    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.824453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824351    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:48.824453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.824392    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\") pod \"da948690-b1a6-4cb1-b51e-c9456d44e39b\" (UID: \"da948690-b1a6-4cb1-b51e-c9456d44e39b\") "
Nov 03 20:31:48.825738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.825709    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.825903 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.825875    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.826068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.826044    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.826178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.826156    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.826261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.826182    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da948690-b1a6-4cb1-b51e-c9456d44e39b-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.827256 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.827234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.828916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.828889    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.828996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.828895    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "kube-api-access-vns7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.828996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.828948    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo" pod "da948690-b1a6-4cb1-b51e-c9456d44e39b" (UID: "da948690-b1a6-4cb1-b51e-c9456d44e39b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:48.846782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.846756    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:31:48.849756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.849727    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:31:48.856152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.856126    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:31:48.927238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927191    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927268    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927326    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927401    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927429    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927475    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927545    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927597    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927629    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927685    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927726    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.927808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927787    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927821    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927853    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.927857    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928018    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928062    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928191    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928220    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928255    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928283    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928333    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928367    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928394    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928452    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928506    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\") pod \"02f07c45-83cc-41f9-bd4c-9b7364e58237\" (UID: \"02f07c45-83cc-41f9-bd4c-9b7364e58237\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928547    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928592    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\") pod \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\" (UID: \"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1\") "
Nov 03 20:31:48.928828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928645    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\") pod \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\" (UID: \"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3\") "
Nov 03 20:31:48.929850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.928874    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.929850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.929725    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:48.931723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.930306    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.931723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.930684    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.931723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.930793    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:48.931723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.931125    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.931723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.931390    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.932239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.932022    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.932429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.932299    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:48.933836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.933811    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.934025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.934004    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.934168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.934143    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.937871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.934193    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.939977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.939951    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f07c45-83cc-41f9-bd4c-9b7364e58237-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.941440 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.940976    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.941870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.941854    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.942740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.942725    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.942843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.942829    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.942949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.942938    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943032    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943112    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943199    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943284    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943390 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943378    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943471 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943460    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943552    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943642    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943727 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943718    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943799    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vns7m\" (UniqueName: \"kubernetes.io/projected/da948690-b1a6-4cb1-b51e-c9456d44e39b-kube-api-access-vns7m\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943898 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943887    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.943985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.943974    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:48.944518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.944496    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.962001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.961939    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.962319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.962284    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:48.962595 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.962515    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "kube-api-access-d6lrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.962820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.962790    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:48.963093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.963063    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token" pod "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" (UID: "2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.963312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.963280    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:48.963397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.963293    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.963785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.963757    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:48.964713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.964679    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "kube-api-access-q2v87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.965128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.965096    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token" pod "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" (UID: "ade2339f-3a0c-47d5-80e3-80f4d66bdbe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:48.966849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:48.966814    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc" pod "02f07c45-83cc-41f9-bd4c-9b7364e58237" (UID: "02f07c45-83cc-41f9-bd4c-9b7364e58237"). InnerVolumeSpecName "kube-api-access-jnnmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:49.045398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045359    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2v87\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-kube-api-access-q2v87\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045400    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045416    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045428    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045440    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045450    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045459    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045468    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnmc\" (UniqueName: \"kubernetes.io/projected/02f07c45-83cc-41f9-bd4c-9b7364e58237-kube-api-access-jnnmc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045478    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045486    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045494    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045501    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lrv\" (UniqueName: \"kubernetes.io/projected/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3-kube-api-access-d6lrv\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.045509    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f07c45-83cc-41f9-bd4c-9b7364e58237-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:49.236053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.236022    5424 generic.go:356] "Generic (PLEG): container finished" podID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerID="b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c" exitCode=0
Nov 03 20:31:49.236255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.236075    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"
Nov 03 20:31:49.236255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.236105    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" event={"ID":"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1","Type":"ContainerDied","Data":"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"}
Nov 03 20:31:49.236255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.236166    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj" event={"ID":"ade2339f-3a0c-47d5-80e3-80f4d66bdbe1","Type":"ContainerDied","Data":"7843f4a27aff751505efd56cacbae9c3932828b5b3f86b68421c8a00b218016b"}
Nov 03 20:31:49.236255 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.236195    5424 scope.go:122] "RemoveContainer" containerID="b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"
Nov 03 20:31:49.238150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.238084    5424 generic.go:356] "Generic (PLEG): container finished" podID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerID="5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669" exitCode=0
Nov 03 20:31:49.238273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.238152    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"
Nov 03 20:31:49.238273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.238152    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" event={"ID":"da948690-b1a6-4cb1-b51e-c9456d44e39b","Type":"ContainerDied","Data":"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"}
Nov 03 20:31:49.238273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.238228    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2" event={"ID":"da948690-b1a6-4cb1-b51e-c9456d44e39b","Type":"ContainerDied","Data":"5b748813d7e38391df5b0e6454d5dadf0f7f4819be64f4c10cc76a71ed83436f"}
Nov 03 20:31:49.243369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.243116    5424 generic.go:356] "Generic (PLEG): container finished" podID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerID="84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc" exitCode=0
Nov 03 20:31:49.243369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.243214    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" event={"ID":"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3","Type":"ContainerDied","Data":"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"}
Nov 03 20:31:49.243369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.243258    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq" event={"ID":"2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3","Type":"ContainerDied","Data":"8751fb769f95c41816440358e50323f1ec3e601cc01a2e32c3214d4c4d39c781"}
Nov 03 20:31:49.243369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.243250    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"
Nov 03 20:31:49.245176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.245155    5424 generic.go:356] "Generic (PLEG): container finished" podID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerID="0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f" exitCode=0
Nov 03 20:31:49.245305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.245287    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" event={"ID":"02f07c45-83cc-41f9-bd4c-9b7364e58237","Type":"ContainerDied","Data":"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"}
Nov 03 20:31:49.245368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.245328    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n" event={"ID":"02f07c45-83cc-41f9-bd4c-9b7364e58237","Type":"ContainerDied","Data":"30c769974a1036af53257915b990bdcc94fb66ce486f5a99b090ec8762c5815c"}
Nov 03 20:31:49.245420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.245383    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"
Nov 03 20:31:49.250924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.250899    5424 scope.go:122] "RemoveContainer" containerID="b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"
Nov 03 20:31:49.251210 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:49.251189    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c\": container with ID starting with b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c not found: ID does not exist" containerID="b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"
Nov 03 20:31:49.251299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.251220    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c"} err="failed to get container status \"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c\": rpc error: code = NotFound desc = could not find container \"b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c\": container with ID starting with b7cec79f868f9b71e7053960c16b0e3a22f4cc287dc546e88d5187fabc0eb35c not found: ID does not exist"
Nov 03 20:31:49.251299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.251237    5424 scope.go:122] "RemoveContainer" containerID="5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"
Nov 03 20:31:49.263043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.263017    5424 scope.go:122] "RemoveContainer" containerID="5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"
Nov 03 20:31:49.263343 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:49.263321    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669\": container with ID starting with 5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669 not found: ID does not exist" containerID="5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"
Nov 03 20:31:49.263414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.263351    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669"} err="failed to get container status \"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669\": rpc error: code = NotFound desc = could not find container \"5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669\": container with ID starting with 5db717b1709c42bf663de8a8a3d10aee7edad9536969d437812a02425dd4b669 not found: ID does not exist"
Nov 03 20:31:49.263414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.263368    5424 scope.go:122] "RemoveContainer" containerID="84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"
Nov 03 20:31:49.273451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.273432    5424 scope.go:122] "RemoveContainer" containerID="84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"
Nov 03 20:31:49.273754 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:49.273731    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc\": container with ID starting with 84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc not found: ID does not exist" containerID="84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"
Nov 03 20:31:49.273815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.273761    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc"} err="failed to get container status \"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc\": rpc error: code = NotFound desc = could not find container \"84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc\": container with ID starting with 84cab34f87efa527225711525466e8478203d2cf81560e11b1a245f69d82b2fc not found: ID does not exist"
Nov 03 20:31:49.273815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.273777    5424 scope.go:122] "RemoveContainer" containerID="0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"
Nov 03 20:31:49.283149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.283129    5424 scope.go:122] "RemoveContainer" containerID="0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"
Nov 03 20:31:49.283428 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:31:49.283410    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f\": container with ID starting with 0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f not found: ID does not exist" containerID="0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"
Nov 03 20:31:49.283494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.283433    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f"} err="failed to get container status \"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f\": rpc error: code = NotFound desc = could not find container \"0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f\": container with ID starting with 0c42fac0375b17d1e7f5aaf5ea4b244bf2621146236d9a0c956527360063507f not found: ID does not exist"
Nov 03 20:31:49.294113 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.294089    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:31:49.306321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.306298    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-5m6p2"]
Nov 03 20:31:49.332982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.332954    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:31:49.336801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.336775    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-sx7fj"]
Nov 03 20:31:49.400234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.400204    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:31:49.404123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.404089    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-8jdkq"]
Nov 03 20:31:49.436785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.436748    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:31:49.445392 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.445364    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-qjf7n"]
Nov 03 20:31:49.921815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.921780    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" path="/var/lib/kubelet/pods/02f07c45-83cc-41f9-bd4c-9b7364e58237/volumes"
Nov 03 20:31:49.922326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.922312    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" path="/var/lib/kubelet/pods/2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3/volumes"
Nov 03 20:31:49.922790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.922776    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" path="/var/lib/kubelet/pods/ade2339f-3a0c-47d5-80e3-80f4d66bdbe1/volumes"
Nov 03 20:31:49.923575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:49.923560    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" path="/var/lib/kubelet/pods/da948690-b1a6-4cb1-b51e-c9456d44e39b/volumes"
Nov 03 20:31:52.481598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.481554    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"]
Nov 03 20:31:52.482151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482137    5424 cpu_manager.go:414] "containerMap: removing container" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy"
Nov 03 20:31:52.482200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482152    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy"
Nov 03 20:31:52.482200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482164    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5d6240bd-4a29-4986-b8d4-b6158da09a5c" containerName="revoke-keys"
Nov 03 20:31:52.482200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482170    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6240bd-4a29-4986-b8d4-b6158da09a5c" containerName="revoke-keys"
Nov 03 20:31:52.482200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482183    5424 cpu_manager.go:414] "containerMap: removing container" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy"
Nov 03 20:31:52.482200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482188    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy"
Nov 03 20:31:52.482347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482215    5424 cpu_manager.go:414] "containerMap: removing container" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy"
Nov 03 20:31:52.482347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482220    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy"
Nov 03 20:31:52.482347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482228    5424 cpu_manager.go:414] "containerMap: removing container" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy"
Nov 03 20:31:52.482347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482233    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy"
Nov 03 20:31:52.482469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482364    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="da948690-b1a6-4cb1-b51e-c9456d44e39b" containerName="istio-proxy"
Nov 03 20:31:52.482469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482379    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5d83c9-8df4-4e40-ade9-4ff9e4f3d3e3" containerName="istio-proxy"
Nov 03 20:31:52.482469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482391    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade2339f-3a0c-47d5-80e3-80f4d66bdbe1" containerName="istio-proxy"
Nov 03 20:31:52.482469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482398    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6240bd-4a29-4986-b8d4-b6158da09a5c" containerName="revoke-keys"
Nov 03 20:31:52.482469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.482404    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f07c45-83cc-41f9-bd4c-9b7364e58237" containerName="istio-proxy"
Nov 03 20:31:52.488013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.487993    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.514769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.514739    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"]
Nov 03 20:31:52.519835 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.519797    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8z6n\" (UniqueName: \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.520034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.519862    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.520097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.520046    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.521512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.521490    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:52.521688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.521669    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:31:52.621259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.621222    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8z6n\" (UniqueName: \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.621473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.621269    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.621473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.621336    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.622043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.622016    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.623978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.623953    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.642506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.642481    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8z6n\" (UniqueName: \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\") pod \"maas-api-revoke-keys-e2e-api-a-fab871-vfb8c\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:52.802370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:52.802334    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:53.111227 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:31:53.111186    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5388b81_2a82_48bb_9930_e2fa6f5662cc.slice/crio-2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270 WatchSource:0}: Error finding container 2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270: Status 404 returned error can't find the container with id 2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270
Nov 03 20:31:53.112584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:53.112553    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"]
Nov 03 20:31:53.388093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:53.388008    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c" event={"ID":"c5388b81-2a82-48bb-9930-e2fa6f5662cc","Type":"ContainerStarted","Data":"2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270"}
Nov 03 20:31:54.089976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.089936    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"]
Nov 03 20:31:54.094384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.094357    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.112388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.112354    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"]
Nov 03 20:31:54.144275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.144242    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.144456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.144333    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlttb\" (UniqueName: \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.144546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.144518    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.246483 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.246443    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.246683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.246606    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.246683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.246658    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlttb\" (UniqueName: \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.247194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.247167    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.249066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.249044    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.267680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.267655    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlttb\" (UniqueName: \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.342041 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.340379    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"]
Nov 03 20:31:54.346696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.346665    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.367127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.367085    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"]
Nov 03 20:31:54.417196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.417160    5424 generic.go:356] "Generic (PLEG): container finished" podID="c5388b81-2a82-48bb-9930-e2fa6f5662cc" containerID="7a4de68ca1fa3e766bfad53a5759c441e14b8c12a169be20f2c6df244981619f" exitCode=0
Nov 03 20:31:54.417400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.417234    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c" event={"ID":"c5388b81-2a82-48bb-9930-e2fa6f5662cc","Type":"ContainerDied","Data":"7a4de68ca1fa3e766bfad53a5759c441e14b8c12a169be20f2c6df244981619f"}
Nov 03 20:31:54.431283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.431251    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:54.448970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.448934    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.449162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.449089    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtszg\" (UniqueName: \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.449316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.449296    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.551986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.550702    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.551986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.550861    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtszg\" (UniqueName: \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.551986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.551142    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.552298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.552057    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.574379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.574337    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.581921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.581889    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtszg\" (UniqueName: \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.730497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.704254    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:54.873063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:54.872853    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"]
Nov 03 20:31:55.127864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.127677    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"]
Nov 03 20:31:55.138971 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:31:55.138941    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd62aa0_11b3_4700_8534_fee9f5922bf8.slice/crio-18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e WatchSource:0}: Error finding container 18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e: Status 404 returned error can't find the container with id 18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e
Nov 03 20:31:55.252195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.252118    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"]
Nov 03 20:31:55.258852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.258824    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.274957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.274914    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76v5t\" (UniqueName: \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.275096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.274987    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.275200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.275184    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.275726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.275706    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"]
Nov 03 20:31:55.376506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.376475    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76v5t\" (UniqueName: \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.376710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.376575    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.376751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.376725    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.377360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.377327    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.379038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.379017    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.398238 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.398203    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76v5t\" (UniqueName: \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\") pod \"maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.428400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.428351    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms" event={"ID":"234cb10e-5114-46b9-abdb-a870bd9cb31e","Type":"ContainerStarted","Data":"f6d4ba09fc01311b7176711186fa6c4f7d1db5f6a68be6bbd7c62d51b51c9918"}
Nov 03 20:31:55.428609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.428408    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms" event={"ID":"234cb10e-5114-46b9-abdb-a870bd9cb31e","Type":"ContainerStarted","Data":"ff5086e77e88f0ac40848cbbaec96570d7d6353c49ceaf65e5eb48bbd1077129"}
Nov 03 20:31:55.429953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.429919    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7" event={"ID":"2bd62aa0-11b3-4700-8534-fee9f5922bf8","Type":"ContainerStarted","Data":"18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e"}
Nov 03 20:31:55.486096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.486037    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms" podStartSLOduration=1.486021453 podStartE2EDuration="1.486021453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:55.485018605 +0000 UTC m=+1657.694034603" watchObservedRunningTime="2026-11-03 20:31:55.486021453 +0000 UTC m=+1657.695037450"
Nov 03 20:31:55.559270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.559240    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:55.575809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.575781    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:55.579053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.579033    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\") pod \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") "
Nov 03 20:31:55.579147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.579123    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\") pod \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") "
Nov 03 20:31:55.579432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.579414    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\" (UniqueName: \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\") pod \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\" (UID: \"c5388b81-2a82-48bb-9930-e2fa6f5662cc\") "
Nov 03 20:31:55.579501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.579435    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca" pod "c5388b81-2a82-48bb-9930-e2fa6f5662cc" (UID: "c5388b81-2a82-48bb-9930-e2fa6f5662cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:55.579962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.579944    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:55.581349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.581328    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca" pod "c5388b81-2a82-48bb-9930-e2fa6f5662cc" (UID: "c5388b81-2a82-48bb-9930-e2fa6f5662cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:55.581670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.581646    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n" pod "c5388b81-2a82-48bb-9930-e2fa6f5662cc" (UID: "c5388b81-2a82-48bb-9930-e2fa6f5662cc"). InnerVolumeSpecName "kube-api-access-n8z6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:55.682568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.681965    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5388b81-2a82-48bb-9930-e2fa6f5662cc-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:55.682568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.681998    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8z6n\" (UniqueName: \"kubernetes.io/projected/c5388b81-2a82-48bb-9930-e2fa6f5662cc-kube-api-access-n8z6n\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:55.914131 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:31:55.914099    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15f66ad_5ca5_409d_bf00_ba59e0fc24f5.slice/crio-57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9 WatchSource:0}: Error finding container 57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9: Status 404 returned error can't find the container with id 57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9
Nov 03 20:31:55.923263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:55.923234    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"]
Nov 03 20:31:56.439347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.439320    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"
Nov 03 20:31:56.439874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.439321    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c" event={"ID":"c5388b81-2a82-48bb-9930-e2fa6f5662cc","Type":"ContainerDied","Data":"2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270"}
Nov 03 20:31:56.439874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.439450    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c63f492bacf10efa8d64352750f21aba223e61d1ae32d50c410db79e92ed270"
Nov 03 20:31:56.441357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.441320    5424 generic.go:356] "Generic (PLEG): container finished" podID="234cb10e-5114-46b9-abdb-a870bd9cb31e" containerID="f6d4ba09fc01311b7176711186fa6c4f7d1db5f6a68be6bbd7c62d51b51c9918" exitCode=0
Nov 03 20:31:56.441478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.441387    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms" event={"ID":"234cb10e-5114-46b9-abdb-a870bd9cb31e","Type":"ContainerDied","Data":"f6d4ba09fc01311b7176711186fa6c4f7d1db5f6a68be6bbd7c62d51b51c9918"}
Nov 03 20:31:56.443511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.443485    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp" event={"ID":"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5","Type":"ContainerStarted","Data":"28b08d1ece9bd011dbe2b2aefdd5c5c609948afc1e95b7aee46a118a03a6636b"}
Nov 03 20:31:56.443661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.443572    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp" event={"ID":"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5","Type":"ContainerStarted","Data":"57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9"}
Nov 03 20:31:56.445117 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.445094    5424 generic.go:356] "Generic (PLEG): container finished" podID="2bd62aa0-11b3-4700-8534-fee9f5922bf8" containerID="128849096f47092b3ba881bacabc9d3b85cb6e5a9ac801e570287caaa9190932" exitCode=0
Nov 03 20:31:56.445208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.445154    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7" event={"ID":"2bd62aa0-11b3-4700-8534-fee9f5922bf8","Type":"ContainerDied","Data":"128849096f47092b3ba881bacabc9d3b85cb6e5a9ac801e570287caaa9190932"}
Nov 03 20:31:56.584983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:56.584913    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp" podStartSLOduration=1.584897566 podStartE2EDuration="1.584897566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:56.583255021 +0000 UTC m=+1658.792271018" watchObservedRunningTime="2026-11-03 20:31:56.584897566 +0000 UTC m=+1658.793913563"
Nov 03 20:31:57.454026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.453988    5424 generic.go:356] "Generic (PLEG): container finished" podID="f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" containerID="28b08d1ece9bd011dbe2b2aefdd5c5c609948afc1e95b7aee46a118a03a6636b" exitCode=0
Nov 03 20:31:57.454447 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.454078    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp" event={"ID":"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5","Type":"ContainerDied","Data":"28b08d1ece9bd011dbe2b2aefdd5c5c609948afc1e95b7aee46a118a03a6636b"}
Nov 03 20:31:57.645476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.645444    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:57.694043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.694004    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:57.717839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.717740    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\" (UniqueName: \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\") pod \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") "
Nov 03 20:31:57.717839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.717806    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\") pod \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") "
Nov 03 20:31:57.718560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.718143    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\") pod \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\" (UID: \"2bd62aa0-11b3-4700-8534-fee9f5922bf8\") "
Nov 03 20:31:57.718856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.718508    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca" pod "2bd62aa0-11b3-4700-8534-fee9f5922bf8" (UID: "2bd62aa0-11b3-4700-8534-fee9f5922bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:57.719545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.719502    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:57.720379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.720352    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca" pod "2bd62aa0-11b3-4700-8534-fee9f5922bf8" (UID: "2bd62aa0-11b3-4700-8534-fee9f5922bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:57.720464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.720430    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg" pod "2bd62aa0-11b3-4700-8534-fee9f5922bf8" (UID: "2bd62aa0-11b3-4700-8534-fee9f5922bf8"). InnerVolumeSpecName "kube-api-access-wtszg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:57.820592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.820517    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\") pod \"234cb10e-5114-46b9-abdb-a870bd9cb31e\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") "
Nov 03 20:31:57.820791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.820776    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\" (UniqueName: \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\") pod \"234cb10e-5114-46b9-abdb-a870bd9cb31e\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") "
Nov 03 20:31:57.820884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.820838    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\") pod \"234cb10e-5114-46b9-abdb-a870bd9cb31e\" (UID: \"234cb10e-5114-46b9-abdb-a870bd9cb31e\") "
Nov 03 20:31:57.821214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.821191    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca" pod "234cb10e-5114-46b9-abdb-a870bd9cb31e" (UID: "234cb10e-5114-46b9-abdb-a870bd9cb31e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:57.821586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.821565    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtszg\" (UniqueName: \"kubernetes.io/projected/2bd62aa0-11b3-4700-8534-fee9f5922bf8-kube-api-access-wtszg\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:57.821653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.821591    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd62aa0-11b3-4700-8534-fee9f5922bf8-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:57.821653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.821607    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:57.822736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.822715    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca" pod "234cb10e-5114-46b9-abdb-a870bd9cb31e" (UID: "234cb10e-5114-46b9-abdb-a870bd9cb31e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:57.822958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.822938    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb" pod "234cb10e-5114-46b9-abdb-a870bd9cb31e" (UID: "234cb10e-5114-46b9-abdb-a870bd9cb31e"). InnerVolumeSpecName "kube-api-access-zlttb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:57.922769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.922742    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlttb\" (UniqueName: \"kubernetes.io/projected/234cb10e-5114-46b9-abdb-a870bd9cb31e-kube-api-access-zlttb\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:57.922956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:57.922913    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/234cb10e-5114-46b9-abdb-a870bd9cb31e-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:58.463257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.463220    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"
Nov 03 20:31:58.463257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.463245    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms" event={"ID":"234cb10e-5114-46b9-abdb-a870bd9cb31e","Type":"ContainerDied","Data":"ff5086e77e88f0ac40848cbbaec96570d7d6353c49ceaf65e5eb48bbd1077129"}
Nov 03 20:31:58.463751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.463319    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5086e77e88f0ac40848cbbaec96570d7d6353c49ceaf65e5eb48bbd1077129"
Nov 03 20:31:58.464889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.464858    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7" event={"ID":"2bd62aa0-11b3-4700-8534-fee9f5922bf8","Type":"ContainerDied","Data":"18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e"}
Nov 03 20:31:58.465020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.464897    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fbdb6c5236b4537c73677cdf5905d8dd75d1d514ca3c535210e2ca7cdeca9e"
Nov 03 20:31:58.465020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.464897    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"
Nov 03 20:31:58.646164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.646135    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:58.743630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.743601    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\") pod \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") "
Nov 03 20:31:58.743845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.743642    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\" (UniqueName: \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\") pod \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") "
Nov 03 20:31:58.743845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.743712    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\") pod \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\" (UID: \"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5\") "
Nov 03 20:31:58.744167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.744131    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca" pod "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" (UID: "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:58.744406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.744389    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:58.745974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.745941    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca" pod "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" (UID: "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:58.746068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.746003    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t" pod "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" (UID: "f15f66ad-5ca5-409d-bf00-ba59e0fc24f5"). InnerVolumeSpecName "kube-api-access-76v5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:58.846015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.845975    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:58.846015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:58.846009    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76v5t\" (UniqueName: \"kubernetes.io/projected/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5-kube-api-access-76v5t\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:31:59.473672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:59.473642    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"
Nov 03 20:31:59.474066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:59.473670    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp" event={"ID":"f15f66ad-5ca5-409d-bf00-ba59e0fc24f5","Type":"ContainerDied","Data":"57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9"}
Nov 03 20:31:59.474066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:31:59.473726    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57bdff5de9321011270177a26978b33b46cc93daf6acd58dc05479bdc6f2d4f9"
Nov 03 20:32:01.537057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:01.537020    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:32:01.537459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:01.537252    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy" containerID="cri-o://4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0" gracePeriod=30
Nov 03 20:32:06.833724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.833692    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:32:06.889283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889254    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889337    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889358    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889434    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889459    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889486    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889505    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889567    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:06.889684 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889576    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.889959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889691    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:06.889959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.889827    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:06.890077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.890032    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\") pod \"0cba4598-49ff-49c1-9afb-072fcce11c09\" (UID: \"0cba4598-49ff-49c1-9afb-072fcce11c09\") "
Nov 03 20:32:06.891131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891103    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:06.891487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891465    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.891622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891495    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.891622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891510    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.891622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891542    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.891867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.891840    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:06.892995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.892317    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:06.892995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.892607    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "kube-api-access-mzbl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:06.892995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.892818    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:06.893581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.893515    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy" pod "0cba4598-49ff-49c1-9afb-072fcce11c09" (UID: "0cba4598-49ff-49c1-9afb-072fcce11c09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:06.992663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.992615    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.992663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.992655    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbl5\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-kube-api-access-mzbl5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.992663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.992668    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.992865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.992681    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cba4598-49ff-49c1-9afb-072fcce11c09-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:06.992865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:06.992694    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cba4598-49ff-49c1-9afb-072fcce11c09-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:07.548161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.548128    5424 generic.go:356] "Generic (PLEG): container finished" podID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerID="4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0" exitCode=0
Nov 03 20:32:07.548376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.548176    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"
Nov 03 20:32:07.548376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.548206    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" event={"ID":"0cba4598-49ff-49c1-9afb-072fcce11c09","Type":"ContainerDied","Data":"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"}
Nov 03 20:32:07.548376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.548267    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn" event={"ID":"0cba4598-49ff-49c1-9afb-072fcce11c09","Type":"ContainerDied","Data":"0e5c38c9ff9a0eda2b9defed620765d108ee8ba4952f92e89380ab54854bfa4f"}
Nov 03 20:32:07.548376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.548299    5424 scope.go:122] "RemoveContainer" containerID="4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"
Nov 03 20:32:07.559477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.559456    5424 scope.go:122] "RemoveContainer" containerID="4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"
Nov 03 20:32:07.559771 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:07.559749    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0\": container with ID starting with 4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0 not found: ID does not exist" containerID="4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"
Nov 03 20:32:07.559881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.559782    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0"} err="failed to get container status \"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0\": rpc error: code = NotFound desc = could not find container \"4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0\": container with ID starting with 4c668d3d21dee18b3b2d2238d4cc036907031e84ec6bf326157a9a9653fc75c0 not found: ID does not exist"
Nov 03 20:32:07.617514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.617491    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:32:07.630694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.630668    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-k4lcn"]
Nov 03 20:32:07.937508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:07.937413    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" path="/var/lib/kubelet/pods/0cba4598-49ff-49c1-9afb-072fcce11c09/volumes"
Nov 03 20:32:11.481446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:11.481411    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"]
Nov 03 20:32:11.496812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:11.496734    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4a71d-2sn9d"]
Nov 03 20:32:11.920721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:11.920682    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705f3fbf-1914-4aa4-97e1-60a4635974cb" path="/var/lib/kubelet/pods/705f3fbf-1914-4aa4-97e1-60a4635974cb/volumes"
Nov 03 20:32:12.286845 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:12.286805    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:32:16.489760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.489703    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:32:16.491235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.491204    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy" containerID="cri-o://a94b660f25291fab9073817b2593a2dd19ea2b068f2bc9eedb51c12b50d4b203" gracePeriod=30
Nov 03 20:32:16.577024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.576988    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:32:16.577252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.577226    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" containerID="cri-o://96152187be0103dd2e84e8ee6597060cf1f5cd7b674ae8227ee8a402d2c80483" gracePeriod=30
Nov 03 20:32:16.633240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.633206    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:32:16.633520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:16.633488    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" containerID="cri-o://216a8a645757492e37352cb95c764e5bfae55f79a8c6b05d6cea31fe25d6ad0b" gracePeriod=30
Nov 03 20:32:18.161977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.161945    5424 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:18.162372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.162006    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:18.407682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.407652    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:32:18.407994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.407949    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy" containerID="cri-o://e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5" gracePeriod=30
Nov 03 20:32:18.429171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.429097    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:32:18.429343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.429300    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy" containerID="cri-o://07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510" gracePeriod=30
Nov 03 20:32:18.784109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.784081    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:32:18.784315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:18.784281    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" containerID="cri-o://436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2" gracePeriod=30
Nov 03 20:32:19.016997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.016963    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"]
Nov 03 20:32:19.017635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017615    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017635    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017669    5424 cpu_manager.go:414] "containerMap: removing container" podUID="2bd62aa0-11b3-4700-8534-fee9f5922bf8" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017677    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd62aa0-11b3-4700-8534-fee9f5922bf8" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017697    5424 cpu_manager.go:414] "containerMap: removing container" podUID="234cb10e-5114-46b9-abdb-a870bd9cb31e" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017705    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234cb10e-5114-46b9-abdb-a870bd9cb31e" containerName="revoke-keys"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017762    5424 cpu_manager.go:414] "containerMap: removing container" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy"
Nov 03 20:32:19.017773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017770    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017780    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c5388b81-2a82-48bb-9930-e2fa6f5662cc" containerName="revoke-keys"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017788    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5388b81-2a82-48bb-9930-e2fa6f5662cc" containerName="revoke-keys"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017945    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" containerName="revoke-keys"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017965    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd62aa0-11b3-4700-8534-fee9f5922bf8" containerName="revoke-keys"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017976    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="234cb10e-5114-46b9-abdb-a870bd9cb31e" containerName="revoke-keys"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.017990    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cba4598-49ff-49c1-9afb-072fcce11c09" containerName="istio-proxy"
Nov 03 20:32:19.018125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.018000    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5388b81-2a82-48bb-9930-e2fa6f5662cc" containerName="revoke-keys"
Nov 03 20:32:19.023432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.023414    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.052653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.052586    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"]
Nov 03 20:32:19.052871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.052855    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:32:19.052975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.052919    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:32:19.112366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.112335    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.112607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.112373    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6q4\" (UniqueName: \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.112607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.112414    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.214038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.214003    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.214038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.214037    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6q4\" (UniqueName: \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.214466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.214063    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.214782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.214761    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.216574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.216517    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.233832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.233808    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6q4\" (UniqueName: \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\") pod \"maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.338231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.338143    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:19.648197 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:32:19.648168    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43359d5a_d8ef_4e98_a400_998191ca09c4.slice/crio-b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb WatchSource:0}: Error finding container b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb: Status 404 returned error can't find the container with id b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb
Nov 03 20:32:19.650458 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.650434    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"]
Nov 03 20:32:19.691018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.690989    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g" event={"ID":"43359d5a-d8ef-4e98-a400-998191ca09c4","Type":"ContainerStarted","Data":"b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb"}
Nov 03 20:32:19.810958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.810926    5424 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:19.811122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:19.810989    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:20.764921 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:20.755710    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g" event={"ID":"43359d5a-d8ef-4e98-a400-998191ca09c4","Type":"ContainerStarted","Data":"f5b1a7579ce7498b6b110a09bd77535a09c567b81d629770fff846bc40d9424a"}
Nov 03 20:32:20.834560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:20.821647    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g" podStartSLOduration=2.821626923 podStartE2EDuration="2.821626923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:20.819697325 +0000 UTC m=+1683.028713329" watchObservedRunningTime="2026-11-03 20:32:20.821626923 +0000 UTC m=+1683.030642920"
Nov 03 20:32:21.150505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.150421    5424 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:21.150505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.150485    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:21.772330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.772281    5424 generic.go:356] "Generic (PLEG): container finished" podID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerID="216a8a645757492e37352cb95c764e5bfae55f79a8c6b05d6cea31fe25d6ad0b" exitCode=0
Nov 03 20:32:21.772845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.772495    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" event={"ID":"52122ad3-9e3d-4c10-86f6-c4ccb9207740","Type":"ContainerDied","Data":"216a8a645757492e37352cb95c764e5bfae55f79a8c6b05d6cea31fe25d6ad0b"}
Nov 03 20:32:21.777584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.774734    5424 generic.go:356] "Generic (PLEG): container finished" podID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerID="96152187be0103dd2e84e8ee6597060cf1f5cd7b674ae8227ee8a402d2c80483" exitCode=0
Nov 03 20:32:21.777584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.774828    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" event={"ID":"02498fcf-c866-4b06-a99f-f0ef7e2f31e3","Type":"ContainerDied","Data":"96152187be0103dd2e84e8ee6597060cf1f5cd7b674ae8227ee8a402d2c80483"}
Nov 03 20:32:21.777584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.777038    5424 generic.go:356] "Generic (PLEG): container finished" podID="43359d5a-d8ef-4e98-a400-998191ca09c4" containerID="f5b1a7579ce7498b6b110a09bd77535a09c567b81d629770fff846bc40d9424a" exitCode=0
Nov 03 20:32:21.777584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.777143    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g" event={"ID":"43359d5a-d8ef-4e98-a400-998191ca09c4","Type":"ContainerDied","Data":"f5b1a7579ce7498b6b110a09bd77535a09c567b81d629770fff846bc40d9424a"}
Nov 03 20:32:21.784253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.779555    5424 generic.go:356] "Generic (PLEG): container finished" podID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerID="a94b660f25291fab9073817b2593a2dd19ea2b068f2bc9eedb51c12b50d4b203" exitCode=0
Nov 03 20:32:21.784253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.779721    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" event={"ID":"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e","Type":"ContainerDied","Data":"a94b660f25291fab9073817b2593a2dd19ea2b068f2bc9eedb51c12b50d4b203"}
Nov 03 20:32:21.784253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.779763    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7" event={"ID":"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e","Type":"ContainerDied","Data":"ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d"}
Nov 03 20:32:21.784253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.779782    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca27ad340384e487b58789487900e537b42c5d99ce7388f0f762e7b0bcd8ef0d"
Nov 03 20:32:21.784253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.783668    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:32:21.793602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793554    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793614    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793656    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793739    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793790    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793843    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.793982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.793903    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.794162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.794014    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.794162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.794089    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\") pod \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\" (UID: \"6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e\") "
Nov 03 20:32:21.795168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.795012    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.795276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.795250    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.795332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.795318    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.795704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.795381    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:21.795704 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.795663    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.797764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.797737    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "kube-api-access-b8rbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:21.797964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.797937    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:21.797964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.797954    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:21.800864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.800843    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy" pod "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" (UID: "6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.896840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896803    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.896840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896843    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rbb\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-kube-api-access-b8rbb\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896860    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896873    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896886    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896899    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896911    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896925    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.897101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.896937    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.925741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.925706    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:32:21.997119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.997078    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:32:21.997892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.997868    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.997940    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.997981    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998026    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998066    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998160    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998242    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998319 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998305    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998380    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\") pod \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\" (UID: \"02498fcf-c866-4b06-a99f-f0ef7e2f31e3\") "
Nov 03 20:32:21.998762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.998739    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.999227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.999200    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:21.999386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.999365    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.999715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.999692    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.999939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:21.999916    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:22.000841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.000501    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.003889 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.003868    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "kube-api-access-mfgbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.004813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.004795    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.004929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.004824    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.006475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.006452    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo" pod "02498fcf-c866-4b06-a99f-f0ef7e2f31e3" (UID: "02498fcf-c866-4b06-a99f-f0ef7e2f31e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:22.100707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100608    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.100707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100674    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.100945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100712    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.100945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100766    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100959    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.100997    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.101186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:22.101186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101141    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101178    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101311    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101352    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\") pod \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\" (UID: \"52122ad3-9e3d-4c10-86f6-c4ccb9207740\") "
Nov 03 20:32:22.101583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101514    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.101809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.101754    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.102177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102154    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.102267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102229    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102249    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102267 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102263    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102275    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102287    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102302    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfgbw\" (UniqueName: \"kubernetes.io/projected/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-kube-api-access-mfgbw\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102315    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102330    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102342    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102354    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102367    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.102484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.102379    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02498fcf-c866-4b06-a99f-f0ef7e2f31e3-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.104196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.104147    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:22.104196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.104149    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.105312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.105287    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.105413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.105333    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx" pod "52122ad3-9e3d-4c10-86f6-c4ccb9207740" (UID: "52122ad3-9e3d-4c10-86f6-c4ccb9207740"). InnerVolumeSpecName "kube-api-access-6r5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.204036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.204002    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.204036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.204030    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.204036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.204041    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.204281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.204052    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5dx\" (UniqueName: \"kubernetes.io/projected/52122ad3-9e3d-4c10-86f6-c4ccb9207740-kube-api-access-6r5dx\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.204281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.204062    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52122ad3-9e3d-4c10-86f6-c4ccb9207740-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:22.788979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.788892    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"
Nov 03 20:32:22.788979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.788916    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg" event={"ID":"02498fcf-c866-4b06-a99f-f0ef7e2f31e3","Type":"ContainerDied","Data":"bdd95304074d66068f85c3ca5f92916d214b8ea0a063d39facf3ccf273a89dec"}
Nov 03 20:32:22.789471 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.788992    5424 scope.go:122] "RemoveContainer" containerID="96152187be0103dd2e84e8ee6597060cf1f5cd7b674ae8227ee8a402d2c80483"
Nov 03 20:32:22.790759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.790738    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"
Nov 03 20:32:22.790759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.790749    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"
Nov 03 20:32:22.790940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.790763    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2" event={"ID":"52122ad3-9e3d-4c10-86f6-c4ccb9207740","Type":"ContainerDied","Data":"6f25255aa329f54df6a31327fa507711982a822270e7e834e2d9d71ac573bbe8"}
Nov 03 20:32:22.803514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.803485    5424 scope.go:122] "RemoveContainer" containerID="216a8a645757492e37352cb95c764e5bfae55f79a8c6b05d6cea31fe25d6ad0b"
Nov 03 20:32:22.858243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.858208    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:32:22.870080 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.870034    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-mqfmg"]
Nov 03 20:32:22.908991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.908959    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:32:22.916346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.916314    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-l54p7"]
Nov 03 20:32:22.956744 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.956697    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:32:22.959033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.959007    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-4pgk2"]
Nov 03 20:32:22.973329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:22.973301    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:23.016694 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.016663    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\") pod \"43359d5a-d8ef-4e98-a400-998191ca09c4\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") "
Nov 03 20:32:23.016892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.016804    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\" (UniqueName: \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\") pod \"43359d5a-d8ef-4e98-a400-998191ca09c4\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") "
Nov 03 20:32:23.016892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.016839    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\") pod \"43359d5a-d8ef-4e98-a400-998191ca09c4\" (UID: \"43359d5a-d8ef-4e98-a400-998191ca09c4\") "
Nov 03 20:32:23.017427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.017394    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca" pod "43359d5a-d8ef-4e98-a400-998191ca09c4" (UID: "43359d5a-d8ef-4e98-a400-998191ca09c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:23.017732 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.017699    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.019511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.019488    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca" pod "43359d5a-d8ef-4e98-a400-998191ca09c4" (UID: "43359d5a-d8ef-4e98-a400-998191ca09c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:23.020183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.020156    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4" pod "43359d5a-d8ef-4e98-a400-998191ca09c4" (UID: "43359d5a-d8ef-4e98-a400-998191ca09c4"). InnerVolumeSpecName "kube-api-access-tk6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:23.119201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.119109    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43359d5a-d8ef-4e98-a400-998191ca09c4-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.119201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.119138    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk6q4\" (UniqueName: \"kubernetes.io/projected/43359d5a-d8ef-4e98-a400-998191ca09c4-kube-api-access-tk6q4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.702098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.702066    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:32:23.728166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728144    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728222    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728246    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728284    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728331    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728396 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728368    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728492    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.728630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728499    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.728630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728570    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.728630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728608    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.729132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.728637    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\") pod \"690d50f6-a2ad-433d-b64c-30e409183661\" (UID: \"690d50f6-a2ad-433d-b64c-30e409183661\") "
Nov 03 20:32:23.729214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729127    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:23.729308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729276    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.729427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729342    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.729506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729466    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690d50f6-a2ad-433d-b64c-30e409183661-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.729506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729479    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.729506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729489    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.729506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729497    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.729506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.729505    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.731022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.730999    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "kube-api-access-8n2h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:23.731143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.731105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:23.731277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.731261    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.731674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.731656    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token" pod "690d50f6-a2ad-433d-b64c-30e409183661" (UID: "690d50f6-a2ad-433d-b64c-30e409183661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:23.775575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.772581    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:32:23.805914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.805649    5424 generic.go:356] "Generic (PLEG): container finished" podID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerID="07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510" exitCode=0
Nov 03 20:32:23.805914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.805751    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"
Nov 03 20:32:23.805914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.805761    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" event={"ID":"8d0c8797-f1c7-4929-a88f-56c0784cd5a0","Type":"ContainerDied","Data":"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"}
Nov 03 20:32:23.805914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.805825    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x" event={"ID":"8d0c8797-f1c7-4929-a88f-56c0784cd5a0","Type":"ContainerDied","Data":"02eb38eb06890388e49a130c4fecf6ae230fc6c2a77777bf4627c1a13a442411"}
Nov 03 20:32:23.805914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.805857    5424 scope.go:122] "RemoveContainer" containerID="07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"
Nov 03 20:32:23.812844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.812754    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g" event={"ID":"43359d5a-d8ef-4e98-a400-998191ca09c4","Type":"ContainerDied","Data":"b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb"}
Nov 03 20:32:23.812844 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.812813    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ab978439be2100b35ab35945751f2ca8655e6ebc1fb21d46f69d9752eddeeb"
Nov 03 20:32:23.814356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.814065    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"
Nov 03 20:32:23.827973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.827940    5424 generic.go:356] "Generic (PLEG): container finished" podID="690d50f6-a2ad-433d-b64c-30e409183661" containerID="e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5" exitCode=0
Nov 03 20:32:23.828213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.828194    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" event={"ID":"690d50f6-a2ad-433d-b64c-30e409183661","Type":"ContainerDied","Data":"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"}
Nov 03 20:32:23.828379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.828364    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77" event={"ID":"690d50f6-a2ad-433d-b64c-30e409183661","Type":"ContainerDied","Data":"f564202824e08fa3503ab48c3f8f6e9842d8a4210f26c0d8cecc51585e3e5d1d"}
Nov 03 20:32:23.828612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.828595    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"
Nov 03 20:32:23.834745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.834714    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.835141 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.835115    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.835740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.835714    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.841803    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.841984    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842357    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842464    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842503    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842596    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842636    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.842787    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\") pod \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\" (UID: \"8d0c8797-f1c7-4929-a88f-56c0784cd5a0\") "
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.843623    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.846039    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.846818    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847707    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847731    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847755    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847771    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847787    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690d50f6-a2ad-433d-b64c-30e409183661-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847802    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2h4\" (UniqueName: \"kubernetes.io/projected/690d50f6-a2ad-433d-b64c-30e409183661-kube-api-access-8n2h4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847824    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690d50f6-a2ad-433d-b64c-30e409183661-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847838    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.848246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.847852    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.860064 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.859998    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:23.861977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.861945    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "kube-api-access-bvwx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:23.863920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.863887    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.865190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.865161    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token" pod "8d0c8797-f1c7-4929-a88f-56c0784cd5a0" (UID: "8d0c8797-f1c7-4929-a88f-56c0784cd5a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:23.905934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.905911    5424 scope.go:122] "RemoveContainer" containerID="07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"
Nov 03 20:32:23.906323 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:23.906291    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510\": container with ID starting with 07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510 not found: ID does not exist" containerID="07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"
Nov 03 20:32:23.906457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.906331    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510"} err="failed to get container status \"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510\": rpc error: code = NotFound desc = could not find container \"07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510\": container with ID starting with 07e171dc93cff9b237d2840d1ce5daf802e7e524a8389672986670529835e510 not found: ID does not exist"
Nov 03 20:32:23.906457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.906350    5424 scope.go:122] "RemoveContainer" containerID="e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"
Nov 03 20:32:23.919407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.918937    5424 scope.go:122] "RemoveContainer" containerID="e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"
Nov 03 20:32:23.919732 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:23.919703    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5\": container with ID starting with e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5 not found: ID does not exist" containerID="e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"
Nov 03 20:32:23.919823 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.919746    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5"} err="failed to get container status \"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5\": rpc error: code = NotFound desc = could not find container \"e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5\": container with ID starting with e8c13ea65e22ed88181c0a6a6428b0666828514ab50f8cfa2b22475d71e5f1a5 not found: ID does not exist"
Nov 03 20:32:23.921188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.921161    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" path="/var/lib/kubelet/pods/02498fcf-c866-4b06-a99f-f0ef7e2f31e3/volumes"
Nov 03 20:32:23.921799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.921779    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" path="/var/lib/kubelet/pods/52122ad3-9e3d-4c10-86f6-c4ccb9207740/volumes"
Nov 03 20:32:23.922318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.922300    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" path="/var/lib/kubelet/pods/6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e/volumes"
Nov 03 20:32:23.937378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.937355    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:32:23.941088 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.941056    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-ppj77"]
Nov 03 20:32:23.949246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.949159    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwx4\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-kube-api-access-bvwx4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.949246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.949195    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.949246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.949211    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:23.949246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:23.949225    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0c8797-f1c7-4929-a88f-56c0784cd5a0-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.049722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.049683    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:32:24.148455 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.148424    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:32:24.151791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.151762    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.151952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.151814    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.151952 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.151839    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152025    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152102    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152179    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152275    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152315    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152420    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152463    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152500    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\") pod \"86d86104-3352-47ae-8f18-1e0b7effee00\" (UID: \"86d86104-3352-47ae-8f18-1e0b7effee00\") "
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152620    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.152812    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153050    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153380    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153494    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153509    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153522    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d86104-3352-47ae-8f18-1e0b7effee00-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.153552    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.154963    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.155002    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "kube-api-access-p8ktk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.155110    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.155674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.155234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo" pod "86d86104-3352-47ae-8f18-1e0b7effee00" (UID: "86d86104-3352-47ae-8f18-1e0b7effee00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:24.157963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.157945    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-lg45x"]
Nov 03 20:32:24.255317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.255291    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d86104-3352-47ae-8f18-1e0b7effee00-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.255317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.255316    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ktk\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-kube-api-access-p8ktk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.255622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.255329    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d86104-3352-47ae-8f18-1e0b7effee00-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.255622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.255339    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d86104-3352-47ae-8f18-1e0b7effee00-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.837485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.837396    5424 generic.go:356] "Generic (PLEG): container finished" podID="86d86104-3352-47ae-8f18-1e0b7effee00" containerID="436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2" exitCode=0
Nov 03 20:32:24.837485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.837451    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"
Nov 03 20:32:24.838027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.837482    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" event={"ID":"86d86104-3352-47ae-8f18-1e0b7effee00","Type":"ContainerDied","Data":"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"}
Nov 03 20:32:24.838027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.837586    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59" event={"ID":"86d86104-3352-47ae-8f18-1e0b7effee00","Type":"ContainerDied","Data":"e256af7d39cefefe5d6ef0313999ac4fea2acfeb5579978b72d1db07ed369475"}
Nov 03 20:32:24.838027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.837622    5424 scope.go:122] "RemoveContainer" containerID="436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"
Nov 03 20:32:24.848505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.848489    5424 scope.go:122] "RemoveContainer" containerID="436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"
Nov 03 20:32:24.848772 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:24.848748    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2\": container with ID starting with 436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2 not found: ID does not exist" containerID="436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"
Nov 03 20:32:24.848830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.848783    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2"} err="failed to get container status \"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2\": rpc error: code = NotFound desc = could not find container \"436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2\": container with ID starting with 436aa18dc494df03d3f959e133ef873d8cfa1585cee613102973bb7f6fadf3c2 not found: ID does not exist"
Nov 03 20:32:24.930867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.930840    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:32:24.943485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:24.943455    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-fdc59"]
Nov 03 20:32:25.922731 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:25.922689    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690d50f6-a2ad-433d-b64c-30e409183661" path="/var/lib/kubelet/pods/690d50f6-a2ad-433d-b64c-30e409183661/volumes"
Nov 03 20:32:25.923271 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:25.923251    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" path="/var/lib/kubelet/pods/86d86104-3352-47ae-8f18-1e0b7effee00/volumes"
Nov 03 20:32:25.923750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:25.923734    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" path="/var/lib/kubelet/pods/8d0c8797-f1c7-4929-a88f-56c0784cd5a0/volumes"
Nov 03 20:32:32.857656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:32.857616    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"]
Nov 03 20:32:32.869438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:32.869396    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-de82c8-ltsm7"]
Nov 03 20:32:33.919771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:33.919740    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd62aa0-11b3-4700-8534-fee9f5922bf8" path="/var/lib/kubelet/pods/2bd62aa0-11b3-4700-8534-fee9f5922bf8/volumes"
Nov 03 20:32:35.100848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:35.100811    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"]
Nov 03 20:32:35.119234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:35.118863    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b917c2-r9cms"]
Nov 03 20:32:35.920972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:35.920941    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234cb10e-5114-46b9-abdb-a870bd9cb31e" path="/var/lib/kubelet/pods/234cb10e-5114-46b9-abdb-a870bd9cb31e/volumes"
Nov 03 20:32:40.320489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.320451    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:32:40.321015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.320830    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy" containerID="cri-o://e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a" gracePeriod=30
Nov 03 20:32:40.865426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865388    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"]
Nov 03 20:32:40.865960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865946    5424 cpu_manager.go:414] "containerMap: removing container" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy"
Nov 03 20:32:40.866004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865962    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy"
Nov 03 20:32:40.866004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865985    5424 cpu_manager.go:414] "containerMap: removing container" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy"
Nov 03 20:32:40.866004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865989    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy"
Nov 03 20:32:40.866004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.865998    5424 cpu_manager.go:414] "containerMap: removing container" podUID="43359d5a-d8ef-4e98-a400-998191ca09c4" containerName="revoke-keys"
Nov 03 20:32:40.866004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866003    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43359d5a-d8ef-4e98-a400-998191ca09c4" containerName="revoke-keys"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866012    5424 cpu_manager.go:414] "containerMap: removing container" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866018    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866032    5424 cpu_manager.go:414] "containerMap: removing container" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866037    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866056    5424 cpu_manager.go:414] "containerMap: removing container" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866063    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866073    5424 cpu_manager.go:414] "containerMap: removing container" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866077    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866164    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="690d50f6-a2ad-433d-b64c-30e409183661" containerName="istio-proxy"
Nov 03 20:32:40.866176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866178    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0c8797-f1c7-4929-a88f-56c0784cd5a0" containerName="istio-proxy"
Nov 03 20:32:40.866466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866184    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="43359d5a-d8ef-4e98-a400-998191ca09c4" containerName="revoke-keys"
Nov 03 20:32:40.866466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866192    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d86104-3352-47ae-8f18-1e0b7effee00" containerName="istio-proxy"
Nov 03 20:32:40.866466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866199    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="52122ad3-9e3d-4c10-86f6-c4ccb9207740" containerName="istio-proxy"
Nov 03 20:32:40.866466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866206    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff7ffa7-fdd6-4dd4-bcdf-a14e098d933e" containerName="istio-proxy"
Nov 03 20:32:40.866466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.866211    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="02498fcf-c866-4b06-a99f-f0ef7e2f31e3" containerName="istio-proxy"
Nov 03 20:32:40.869586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.869569    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:40.886248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.886216    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"]
Nov 03 20:32:40.925134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.925087    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:40.925354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.925190    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:40.925354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.925225    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7nw\" (UniqueName: \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:40.932199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.932156    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:32:40.932199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:40.932190    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:32:41.026338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.026300    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.026499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.026348    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.026499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.026389    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7nw\" (UniqueName: \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.027050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.027028    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.029008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.028984    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.045644 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.045605    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7nw\" (UniqueName: \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.184048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.183958    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:41.497915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.497863    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"]
Nov 03 20:32:41.504520 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:32:41.501051    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63e36e3_033f_47c4_8762_ffbbe2d0afb7.slice/crio-4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663 WatchSource:0}: Error finding container 4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663: Status 404 returned error can't find the container with id 4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663
Nov 03 20:32:41.976757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.976715    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr" event={"ID":"b63e36e3-033f-47c4-8762-ffbbe2d0afb7","Type":"ContainerStarted","Data":"340d3541a1e6be6a2985b17ff377e5df128ecb2d1366b8c1b8668fb4387ba6c5"}
Nov 03 20:32:41.976975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:41.976781    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr" event={"ID":"b63e36e3-033f-47c4-8762-ffbbe2d0afb7","Type":"ContainerStarted","Data":"4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663"}
Nov 03 20:32:42.020577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.016762    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:32:42.020577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.017035    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy" containerID="cri-o://41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6" gracePeriod=30
Nov 03 20:32:42.101034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.100983    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr" podStartSLOduration=2.100965496 podStartE2EDuration="2.100965496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:42.02633935 +0000 UTC m=+1704.235355348" watchObservedRunningTime="2026-11-03 20:32:42.100965496 +0000 UTC m=+1704.309981492"
Nov 03 20:32:42.295083 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:42.295045    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:32:42.501180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.501146    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"]
Nov 03 20:32:42.505753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.505716    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fab871-vfb8c"]
Nov 03 20:32:42.634385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.634296    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"]
Nov 03 20:32:42.639206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.639176    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.652922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.652879    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"]
Nov 03 20:32:42.778636 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.778598    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsr8b\" (UniqueName: \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.778813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.778750    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.778912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.778894    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.880767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.880711    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.880938 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.880872    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsr8b\" (UniqueName: \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.880976 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.880937    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.881391 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.881362    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.883429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.883402    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.900036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.899983    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsr8b\" (UniqueName: \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:42.966836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:42.966800    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:43.050273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:43.050235    5424 generic.go:356] "Generic (PLEG): container finished" podID="b63e36e3-033f-47c4-8762-ffbbe2d0afb7" containerID="340d3541a1e6be6a2985b17ff377e5df128ecb2d1366b8c1b8668fb4387ba6c5" exitCode=0
Nov 03 20:32:43.050579 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:43.050357    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr" event={"ID":"b63e36e3-033f-47c4-8762-ffbbe2d0afb7","Type":"ContainerDied","Data":"340d3541a1e6be6a2985b17ff377e5df128ecb2d1366b8c1b8668fb4387ba6c5"}
Nov 03 20:32:43.322639 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:32:43.322602    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda918578e_e771_48fa_b536_73e65fe85280.slice/crio-b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8 WatchSource:0}: Error finding container b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8: Status 404 returned error can't find the container with id b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8
Nov 03 20:32:43.322747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:43.322643    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"]
Nov 03 20:32:43.920690 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:43.920649    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5388b81-2a82-48bb-9930-e2fa6f5662cc" path="/var/lib/kubelet/pods/c5388b81-2a82-48bb-9930-e2fa6f5662cc/volumes"
Nov 03 20:32:44.059753 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.059712    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g" event={"ID":"a918578e-e771-48fa-b536-73e65fe85280","Type":"ContainerStarted","Data":"7472554314a2e95a23997b8ca67f36a315f67c668bd4f5c0ba63690aff918642"}
Nov 03 20:32:44.059919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.059780    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g" event={"ID":"a918578e-e771-48fa-b536-73e65fe85280","Type":"ContainerStarted","Data":"b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8"}
Nov 03 20:32:44.110594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.108899    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g" podStartSLOduration=2.108876412 podStartE2EDuration="2.108876412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:44.106700229 +0000 UTC m=+1706.315716230" watchObservedRunningTime="2026-11-03 20:32:44.108876412 +0000 UTC m=+1706.317892411"
Nov 03 20:32:44.244021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.243992    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:44.410758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.410658    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\" (UniqueName: \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\") pod \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") "
Nov 03 20:32:44.410758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.410721    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\") pod \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") "
Nov 03 20:32:44.410758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.410744    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\") pod \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\" (UID: \"b63e36e3-033f-47c4-8762-ffbbe2d0afb7\") "
Nov 03 20:32:44.411185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.411156    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca" pod "b63e36e3-033f-47c4-8762-ffbbe2d0afb7" (UID: "b63e36e3-033f-47c4-8762-ffbbe2d0afb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:44.411359 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.411343    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:44.413010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.412980    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca" pod "b63e36e3-033f-47c4-8762-ffbbe2d0afb7" (UID: "b63e36e3-033f-47c4-8762-ffbbe2d0afb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:44.413114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.413045    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw" pod "b63e36e3-033f-47c4-8762-ffbbe2d0afb7" (UID: "b63e36e3-033f-47c4-8762-ffbbe2d0afb7"). InnerVolumeSpecName "kube-api-access-ch7nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:44.512720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.512681    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7nw\" (UniqueName: \"kubernetes.io/projected/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-kube-api-access-ch7nw\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:44.512720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:44.512717    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b63e36e3-033f-47c4-8762-ffbbe2d0afb7-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.069356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.069320    5424 generic.go:356] "Generic (PLEG): container finished" podID="a918578e-e771-48fa-b536-73e65fe85280" containerID="7472554314a2e95a23997b8ca67f36a315f67c668bd4f5c0ba63690aff918642" exitCode=0
Nov 03 20:32:45.069861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.069415    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g" event={"ID":"a918578e-e771-48fa-b536-73e65fe85280","Type":"ContainerDied","Data":"7472554314a2e95a23997b8ca67f36a315f67c668bd4f5c0ba63690aff918642"}
Nov 03 20:32:45.071109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.071086    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"
Nov 03 20:32:45.071239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.071108    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr" event={"ID":"b63e36e3-033f-47c4-8762-ffbbe2d0afb7","Type":"ContainerDied","Data":"4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663"}
Nov 03 20:32:45.071239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.071152    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6245b7377fb5053f0b936b973c5a6e89f1472bd2d2cb31bc3c0f61e9123663"
Nov 03 20:32:45.597161 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.597100    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:32:45.729247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729212    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729324    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729387    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729502    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729506    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:45.729651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729549    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729639    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:45.729882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729649    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729734    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.729882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729876    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.730083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729930    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\") pod \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\" (UID: \"1ff940dd-11f0-44c2-8c65-51041e56fcd7\") "
Nov 03 20:32:45.730083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729961    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:45.730083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.729969    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:45.730245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730139    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:45.730574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730521    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.730679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730578    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.730679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730593    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.730679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730606    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.730679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.730619    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.732090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.732062    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:45.732336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.732309    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:45.732435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.732364    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "kube-api-access-mvs4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:45.732435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.732383    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token" pod "1ff940dd-11f0-44c2-8c65-51041e56fcd7" (UID: "1ff940dd-11f0-44c2-8c65-51041e56fcd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:45.832007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.831962    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.832189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.832015    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.832189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.832030    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ff940dd-11f0-44c2-8c65-51041e56fcd7-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:45.832189 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:45.832044    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvs4q\" (UniqueName: \"kubernetes.io/projected/1ff940dd-11f0-44c2-8c65-51041e56fcd7-kube-api-access-mvs4q\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:46.080213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.080178    5424 generic.go:356] "Generic (PLEG): container finished" podID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerID="e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a" exitCode=0
Nov 03 20:32:46.080671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.080250    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"
Nov 03 20:32:46.080671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.080260    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerDied","Data":"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"}
Nov 03 20:32:46.080671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.080323    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc" event={"ID":"1ff940dd-11f0-44c2-8c65-51041e56fcd7","Type":"ContainerDied","Data":"c384c79588d3fbc6b40e2a6848f9deb6ea48fdc3a9db9ac23385afcf2c6e4b9b"}
Nov 03 20:32:46.080671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.080354    5424 scope.go:122] "RemoveContainer" containerID="e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"
Nov 03 20:32:46.091156 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.091143    5424 scope.go:122] "RemoveContainer" containerID="aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"
Nov 03 20:32:46.101442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.101420    5424 scope.go:122] "RemoveContainer" containerID="e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"
Nov 03 20:32:46.101797 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:46.101774    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a\": container with ID starting with e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a not found: ID does not exist" containerID="e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"
Nov 03 20:32:46.101881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.101811    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a"} err="failed to get container status \"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a\": rpc error: code = NotFound desc = could not find container \"e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a\": container with ID starting with e358d58f6a954d52b8977e7d4d625dae4275382e23c881167d13de197a60595a not found: ID does not exist"
Nov 03 20:32:46.101881 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.101836    5424 scope.go:122] "RemoveContainer" containerID="aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"
Nov 03 20:32:46.102123 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:46.102101    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a\": container with ID starting with aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a not found: ID does not exist" containerID="aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"
Nov 03 20:32:46.102162 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.102130    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a"} err="failed to get container status \"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a\": rpc error: code = NotFound desc = could not find container \"aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a\": container with ID starting with aa9869a5a5898a80f99041531efb1025356472c9dcc17c8f7b518b323f38861a not found: ID does not exist"
Nov 03 20:32:46.138018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.137987    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:32:46.149675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.149644    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-de82c8-openshift-default-54fd788897-mskjc"]
Nov 03 20:32:46.231501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.231474    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:46.337975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.337925    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\") pod \"a918578e-e771-48fa-b536-73e65fe85280\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") "
Nov 03 20:32:46.338148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.338023    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\" (UniqueName: \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\") pod \"a918578e-e771-48fa-b536-73e65fe85280\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") "
Nov 03 20:32:46.338148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.338103    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\") pod \"a918578e-e771-48fa-b536-73e65fe85280\" (UID: \"a918578e-e771-48fa-b536-73e65fe85280\") "
Nov 03 20:32:46.338866 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.338773    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca" pod "a918578e-e771-48fa-b536-73e65fe85280" (UID: "a918578e-e771-48fa-b536-73e65fe85280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:46.341045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.341019    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca" pod "a918578e-e771-48fa-b536-73e65fe85280" (UID: "a918578e-e771-48fa-b536-73e65fe85280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:46.341045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.341027    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b" pod "a918578e-e771-48fa-b536-73e65fe85280" (UID: "a918578e-e771-48fa-b536-73e65fe85280"). InnerVolumeSpecName "kube-api-access-fsr8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:46.440432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.440396    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a918578e-e771-48fa-b536-73e65fe85280-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:46.440432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.440426    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a918578e-e771-48fa-b536-73e65fe85280-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:46.440432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:46.440436    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsr8b\" (UniqueName: \"kubernetes.io/projected/a918578e-e771-48fa-b536-73e65fe85280-kube-api-access-fsr8b\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.092063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.092038    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"
Nov 03 20:32:47.092469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.092059    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g" event={"ID":"a918578e-e771-48fa-b536-73e65fe85280","Type":"ContainerDied","Data":"b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8"}
Nov 03 20:32:47.092469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.092110    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4429cdbd506206c235ef0fc7fb1d8d7641010a3078d1eb41d8cba45cb7fb3b8"
Nov 03 20:32:47.342742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.342678    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:32:47.453403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453365    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453510    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453572    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453607    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453659    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453717    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:47.453805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453757    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.453955 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453851    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.454015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453930    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:47.454015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.453958    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.454128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454054    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\") pod \"018e100d-188f-4968-ad37-869b18b33f59\" (UID: \"018e100d-188f-4968-ad37-869b18b33f59\") "
Nov 03 20:32:47.454128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454066    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:47.454235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454188    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:47.454386 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454365    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:47.454752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454634    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.454752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454655    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018e100d-188f-4968-ad37-869b18b33f59-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.454752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454668    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.454752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454679    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.454752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.454688    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.456269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.456234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:47.456384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.456360    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "kube-api-access-qjhw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:47.456462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.456441    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:47.456499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.456469    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy" pod "018e100d-188f-4968-ad37-869b18b33f59" (UID: "018e100d-188f-4968-ad37-869b18b33f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:47.556308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.556270    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018e100d-188f-4968-ad37-869b18b33f59-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.556308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.556300    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018e100d-188f-4968-ad37-869b18b33f59-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.556308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.556310    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhw9\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-kube-api-access-qjhw9\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.556598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.556321    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018e100d-188f-4968-ad37-869b18b33f59-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:47.923715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:47.923662    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" path="/var/lib/kubelet/pods/1ff940dd-11f0-44c2-8c65-51041e56fcd7/volumes"
Nov 03 20:32:48.102906 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.102865    5424 generic.go:356] "Generic (PLEG): container finished" podID="018e100d-188f-4968-ad37-869b18b33f59" containerID="41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6" exitCode=0
Nov 03 20:32:48.103367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.102927    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"
Nov 03 20:32:48.103367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.102953    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerDied","Data":"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"}
Nov 03 20:32:48.103367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.103013    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq" event={"ID":"018e100d-188f-4968-ad37-869b18b33f59","Type":"ContainerDied","Data":"6d0ab02b5874f93ca1d0005a6ed23fa5b7f3f1c7932675c4c0bcc07de88bb84c"}
Nov 03 20:32:48.103367 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.103038    5424 scope.go:122] "RemoveContainer" containerID="41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"
Nov 03 20:32:48.125254 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.124203    5424 scope.go:122] "RemoveContainer" containerID="8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"
Nov 03 20:32:48.135153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.135127    5424 scope.go:122] "RemoveContainer" containerID="41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"
Nov 03 20:32:48.135463 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:48.135442    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6\": container with ID starting with 41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6 not found: ID does not exist" containerID="41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"
Nov 03 20:32:48.135572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.135475    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6"} err="failed to get container status \"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6\": rpc error: code = NotFound desc = could not find container \"41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6\": container with ID starting with 41ca5e5c0a460a635e87d2317f787dbde8b26b9f30ef7330c547e781eb0fadf6 not found: ID does not exist"
Nov 03 20:32:48.135572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.135494    5424 scope.go:122] "RemoveContainer" containerID="8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"
Nov 03 20:32:48.135805 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:48.135789    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10\": container with ID starting with 8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10 not found: ID does not exist" containerID="8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"
Nov 03 20:32:48.135850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.135810    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10"} err="failed to get container status \"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10\": rpc error: code = NotFound desc = could not find container \"8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10\": container with ID starting with 8afb32f5db0205e698835c782053e06c8eafc13a1847ae5802db10cc6f0b0f10 not found: ID does not exist"
Nov 03 20:32:48.165988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.165957    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:32:48.178476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:48.178390    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b917c2-openshift-default-786f669d99-hf6qq"]
Nov 03 20:32:49.929240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:49.929202    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018e100d-188f-4968-ad37-869b18b33f59" path="/var/lib/kubelet/pods/018e100d-188f-4968-ad37-869b18b33f59/volumes"
Nov 03 20:32:50.292669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.292629    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"]
Nov 03 20:32:50.293218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293199    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b63e36e3-033f-47c4-8762-ffbbe2d0afb7" containerName="revoke-keys"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293220    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63e36e3-033f-47c4-8762-ffbbe2d0afb7" containerName="revoke-keys"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293256    5424 cpu_manager.go:414] "containerMap: removing container" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293264    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293282    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293289    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293302    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a918578e-e771-48fa-b536-73e65fe85280" containerName="revoke-keys"
Nov 03 20:32:50.293329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293309    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a918578e-e771-48fa-b536-73e65fe85280" containerName="revoke-keys"
Nov 03 20:32:50.293728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293494    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:50.293728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293513    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63e36e3-033f-47c4-8762-ffbbe2d0afb7" containerName="revoke-keys"
Nov 03 20:32:50.293728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293552    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:50.293728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293561    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a918578e-e771-48fa-b536-73e65fe85280" containerName="revoke-keys"
Nov 03 20:32:50.293728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.293570    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:50.298803 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.298782    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.314243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.314213    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"]
Nov 03 20:32:50.322792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.322689    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:32:50.322792 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.322689    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:32:50.391771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.391739    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.391971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.391786    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.392040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.392022    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmxc\" (UniqueName: \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.493217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.493182    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.493409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.493228    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.493409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.493316    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmxc\" (UniqueName: \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.494026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.493997    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.496038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.496014    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.512799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.512775    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmxc\" (UniqueName: \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\") pod \"maas-api-revoke-keys-e2e-api-b-d05604-4dbcq\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.613826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.613735    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:50.921126 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:32:50.921097    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faf616e_5358_4364_8c68_97d8363b2e96.slice/crio-efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d WatchSource:0}: Error finding container efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d: Status 404 returned error can't find the container with id efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d
Nov 03 20:32:50.922699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:50.922674    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"]
Nov 03 20:32:51.135401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.135370    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq" event={"ID":"5faf616e-5358-4364-8c68-97d8363b2e96","Type":"ContainerStarted","Data":"efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d"}
Nov 03 20:32:51.183365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.183282    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:32:51.184134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.184112    5424 cpu_manager.go:414] "containerMap: removing container" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:51.184182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.184140    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:51.184182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.184158    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:51.184182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.184166    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff940dd-11f0-44c2-8c65-51041e56fcd7" containerName="istio-proxy"
Nov 03 20:32:51.184387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.184368    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="018e100d-188f-4968-ad37-869b18b33f59" containerName="istio-proxy"
Nov 03 20:32:51.189356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.189332    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.204024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.203994    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmgr\" (UniqueName: \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.204174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.204101    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.204174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.204151    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.204360 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.204335    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.207399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.207311    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:32:51.214615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.214585    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1d159a-serving-cert\""
Nov 03 20:32:51.305841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.305811    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmgr\" (UniqueName: \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.306014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.305863    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.306014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.305911    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.306014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.305977    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.306142 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:51.306041    5424 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-serving-cert: secret "maas-api-e2e-inf-b-1d159a-serving-cert" not found
Nov 03 20:32:51.306142 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:51.306130    5424 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls podName:13fc3847-b72e-4bcd-b8cc-739de4da5979 nodeName:}" failed. No retries permitted until 2026-11-03 20:32:51.806109675 +0000 UTC m=+1714.015125669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls") pod "maas-api-e2e-inf-b-1d159a-757c897574-5m94c" (UID: "13fc3847-b72e-4bcd-b8cc-739de4da5979") : secret "maas-api-e2e-inf-b-1d159a-serving-cert" not found
Nov 03 20:32:51.306661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.306637    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.308494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.308474    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.326164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.326137    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmgr\" (UniqueName: \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.661342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.661306    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:32:51.815311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.815272    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:51.817904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:51.817872    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") pod \"maas-api-e2e-inf-b-1d159a-757c897574-5m94c\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:52.106201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:52.106166    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:52.157128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:52.156289    5424 generic.go:356] "Generic (PLEG): container finished" podID="5faf616e-5358-4364-8c68-97d8363b2e96" containerID="d6b98b2f9b11113721666527c371a98e7ff1583f3caf4d52a859078ee48ed33f" exitCode=0
Nov 03 20:32:52.157128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:52.156616    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq" event={"ID":"5faf616e-5358-4364-8c68-97d8363b2e96","Type":"ContainerDied","Data":"d6b98b2f9b11113721666527c371a98e7ff1583f3caf4d52a859078ee48ed33f"}
Nov 03 20:32:52.457054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:52.456949    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:32:52.494815 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:32:52.493352    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fc3847_b72e_4bcd_b8cc_739de4da5979.slice/crio-1406989327f7e08588add7cb30cd35150ed6519e43f1b3d3dafc1691a20de9c5 WatchSource:0}: Error finding container 1406989327f7e08588add7cb30cd35150ed6519e43f1b3d3dafc1691a20de9c5: Status 404 returned error can't find the container with id 1406989327f7e08588add7cb30cd35150ed6519e43f1b3d3dafc1691a20de9c5
Nov 03 20:32:52.500550 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:52.496273    5424 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:32:53.177612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.171727    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" event={"ID":"13fc3847-b72e-4bcd-b8cc-739de4da5979","Type":"ContainerStarted","Data":"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"}
Nov 03 20:32:53.177612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.171790    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" event={"ID":"13fc3847-b72e-4bcd-b8cc-739de4da5979","Type":"ContainerStarted","Data":"1406989327f7e08588add7cb30cd35150ed6519e43f1b3d3dafc1691a20de9c5"}
Nov 03 20:32:53.177612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.171834    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:32:53.245552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.244508    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" podStartSLOduration=1.946785014 podStartE2EDuration="2.244487713s" totalImagesPullingTime="297.702699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:53.239578684 +0000 UTC m=+1715.448594684" watchObservedRunningTime="2026-11-03 20:32:53.244487713 +0000 UTC m=+1715.453503711"
Nov 03 20:32:53.387134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.387105    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:53.448643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.448517    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\") pod \"5faf616e-5358-4364-8c68-97d8363b2e96\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") "
Nov 03 20:32:53.448972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.448644    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\") pod \"5faf616e-5358-4364-8c68-97d8363b2e96\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") "
Nov 03 20:32:53.449641 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.448787    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\" (UniqueName: \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\") pod \"5faf616e-5358-4364-8c68-97d8363b2e96\" (UID: \"5faf616e-5358-4364-8c68-97d8363b2e96\") "
Nov 03 20:32:53.452235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.452188    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca" pod "5faf616e-5358-4364-8c68-97d8363b2e96" (UID: "5faf616e-5358-4364-8c68-97d8363b2e96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:53.459945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.459916    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc" pod "5faf616e-5358-4364-8c68-97d8363b2e96" (UID: "5faf616e-5358-4364-8c68-97d8363b2e96"). InnerVolumeSpecName "kube-api-access-djmxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:53.460075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.459985    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca" pod "5faf616e-5358-4364-8c68-97d8363b2e96" (UID: "5faf616e-5358-4364-8c68-97d8363b2e96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:53.555557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.551997    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmxc\" (UniqueName: \"kubernetes.io/projected/5faf616e-5358-4364-8c68-97d8363b2e96-kube-api-access-djmxc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:53.555557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.552037    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:53.555557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:53.552054    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5faf616e-5358-4364-8c68-97d8363b2e96-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:54.184552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.183764    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"
Nov 03 20:32:54.184552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.183791    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq" event={"ID":"5faf616e-5358-4364-8c68-97d8363b2e96","Type":"ContainerDied","Data":"efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d"}
Nov 03 20:32:54.184552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.183857    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd8f7204883bdedcc6b55af17c851330aa971370e224fe15fceeb308a08fd1d"
Nov 03 20:32:54.557011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.556910    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"]
Nov 03 20:32:54.560575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.560523    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d1aa66-kp6dp"]
Nov 03 20:32:54.863915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.863816    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"]
Nov 03 20:32:54.867414 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:54.867389    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-17b3d9-nnl2g"]
Nov 03 20:32:55.923842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:55.923795    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a918578e-e771-48fa-b536-73e65fe85280" path="/var/lib/kubelet/pods/a918578e-e771-48fa-b536-73e65fe85280/volumes"
Nov 03 20:32:55.924314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:55.924298    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15f66ad-5ca5-409d-bf00-ba59e0fc24f5" path="/var/lib/kubelet/pods/f15f66ad-5ca5-409d-bf00-ba59e0fc24f5/volumes"
Nov 03 20:32:56.662439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:56.662366    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" podUID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" containerName="payload-pre-processing" containerID="cri-o://efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f" gracePeriod=25
Nov 03 20:32:56.937968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:56.937943    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:32:57.085828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.085750    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\") pod \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") "
Nov 03 20:32:57.085828 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.085790    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\" (UniqueName: \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\") pod \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\" (UID: \"9077b4af-e665-4e50-b5d6-eff8cdb2e839\") "
Nov 03 20:32:57.086164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.086140    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume" pod "9077b4af-e665-4e50-b5d6-eff8cdb2e839" (UID: "9077b4af-e665-4e50-b5d6-eff8cdb2e839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:57.086263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.086247    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9077b4af-e665-4e50-b5d6-eff8cdb2e839-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:57.088155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.088134    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt" pod "9077b4af-e665-4e50-b5d6-eff8cdb2e839" (UID: "9077b4af-e665-4e50-b5d6-eff8cdb2e839"). InnerVolumeSpecName "kube-api-access-h88kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:57.187904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.187816    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88kt\" (UniqueName: \"kubernetes.io/projected/9077b4af-e665-4e50-b5d6-eff8cdb2e839-kube-api-access-h88kt\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:32:57.207894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.207861    5424 generic.go:356] "Generic (PLEG): container finished" podID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" containerID="efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f" exitCode=0
Nov 03 20:32:57.208085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.207907    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"
Nov 03 20:32:57.208085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.207935    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" event={"ID":"9077b4af-e665-4e50-b5d6-eff8cdb2e839","Type":"ContainerDied","Data":"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"}
Nov 03 20:32:57.208085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.207998    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k" event={"ID":"9077b4af-e665-4e50-b5d6-eff8cdb2e839","Type":"ContainerDied","Data":"8bd2432fd0f1785fdd83f0f62b5867baab5e893b7b964351c875bbb4b3c2e275"}
Nov 03 20:32:57.208085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.208021    5424 scope.go:122] "RemoveContainer" containerID="efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"
Nov 03 20:32:57.218807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.218785    5424 scope.go:122] "RemoveContainer" containerID="efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"
Nov 03 20:32:57.219100 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:32:57.219077    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f\": container with ID starting with efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f not found: ID does not exist" containerID="efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"
Nov 03 20:32:57.219150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.219111    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f"} err="failed to get container status \"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f\": rpc error: code = NotFound desc = could not find container \"efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f\": container with ID starting with efbfa3a74531cf747f75a9001f92bdfba4c4f8984cf4c8d97de8980689aab51f not found: ID does not exist"
Nov 03 20:32:57.261504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.261472    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:32:57.273972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.273939    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-e3ba87-6859c55dd7-xpj9k"]
Nov 03 20:32:57.922031 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:57.921998    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" path="/var/lib/kubelet/pods/9077b4af-e665-4e50-b5d6-eff8cdb2e839/volumes"
Nov 03 20:32:59.190132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:32:59.190102    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:33:01.035357 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:01.035320    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"]
Nov 03 20:33:01.045165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:01.045137    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b5b66-lnmq9"]
Nov 03 20:33:01.920411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:01.920377    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12996b4-4c39-4a6d-ad6f-ae84a8e3543e" path="/var/lib/kubelet/pods/c12996b4-4c39-4a6d-ad6f-ae84a8e3543e/volumes"
Nov 03 20:33:02.825612 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:02.825570    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:33:02.826046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:02.825861    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" containerID="cri-o://c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3" gracePeriod=30
Nov 03 20:33:03.642033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.641999    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"]
Nov 03 20:33:03.642657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642635    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" containerName="payload-pre-processing"
Nov 03 20:33:03.642657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642656    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" containerName="payload-pre-processing"
Nov 03 20:33:03.642860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642676    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5faf616e-5358-4364-8c68-97d8363b2e96" containerName="revoke-keys"
Nov 03 20:33:03.642860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642683    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faf616e-5358-4364-8c68-97d8363b2e96" containerName="revoke-keys"
Nov 03 20:33:03.642860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642817    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9077b4af-e665-4e50-b5d6-eff8cdb2e839" containerName="payload-pre-processing"
Nov 03 20:33:03.642860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.642832    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faf616e-5358-4364-8c68-97d8363b2e96" containerName="revoke-keys"
Nov 03 20:33:03.647871 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.647849    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.663945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.663911    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:33:03.664084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.663965    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:33:03.664084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.663968    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"]
Nov 03 20:33:03.675085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.675053    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.675241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.675119    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.675352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.675330    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkzm\" (UniqueName: \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.777253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.777217    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkzm\" (UniqueName: \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.777453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.777343    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.777453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.777379    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.778032 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.777997    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.779912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.779888    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.796670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.796638    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkzm\" (UniqueName: \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\") pod \"maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:03.962230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:03.962136    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:04.273333 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:04.272355    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5a8e6e_2d2b_4587_bd56_1ba27305ed6b.slice/crio-2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02 WatchSource:0}: Error finding container 2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02: Status 404 returned error can't find the container with id 2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02
Nov 03 20:33:04.274068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:04.274041    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"]
Nov 03 20:33:04.559020 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:04.558979    5424 patch_prober.go:29] interesting pod/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:33:04.559183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:04.559046    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:33:05.271799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:05.271759    5424 generic.go:356] "Generic (PLEG): container finished" podID="5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" containerID="ac7fe97fee657b7a05d9cbe111221e782ab6e41c1b8d420549eab6b25e50bd75" exitCode=0
Nov 03 20:33:05.272200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:05.271840    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls" event={"ID":"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b","Type":"ContainerDied","Data":"ac7fe97fee657b7a05d9cbe111221e782ab6e41c1b8d420549eab6b25e50bd75"}
Nov 03 20:33:05.272200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:05.271905    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls" event={"ID":"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b","Type":"ContainerStarted","Data":"2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02"}
Nov 03 20:33:06.451834 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.451805    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:06.516326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.516293    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\") pod \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") "
Nov 03 20:33:06.516522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.516351    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\") pod \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") "
Nov 03 20:33:06.516655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.516634    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\" (UniqueName: \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\") pod \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\" (UID: \"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b\") "
Nov 03 20:33:06.516725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.516700    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca" pod "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" (UID: "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:06.517168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.517148    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.518756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.518734    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca" pod "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" (UID: "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:06.518951 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.518928    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm" pod "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" (UID: "5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b"). InnerVolumeSpecName "kube-api-access-sxkzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:06.619078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.618975    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxkzm\" (UniqueName: \"kubernetes.io/projected/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-kube-api-access-sxkzm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.619078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:06.619013    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:07.287740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:07.287713    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"
Nov 03 20:33:07.287932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:07.287753    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls" event={"ID":"5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b","Type":"ContainerDied","Data":"2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02"}
Nov 03 20:33:07.287932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:07.287799    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8e4778b36a0f0f3656d50913c8e021725d909685ad48932643d82690e63f02"
Nov 03 20:33:08.098332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.098264    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:33:08.136857 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.136822    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.136914    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.136948    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.136997    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137114    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137172    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137225    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137217    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.137310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137278    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137288    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.137520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137314    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\") pod \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\" (UID: \"c8a1ff7d-4d50-4776-ac88-70ab88414d78\") "
Nov 03 20:33:08.137618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137566    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.137618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137570    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:08.137840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.137816    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.138157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.138136    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.138251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.138162    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.138251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.138175    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.138251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.138190    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.138251 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.138203    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.139622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.139594    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.139710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.139622    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:08.139902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.139884    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:08.140118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.140097    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq" pod "c8a1ff7d-4d50-4776-ac88-70ab88414d78" (UID: "c8a1ff7d-4d50-4776-ac88-70ab88414d78"). InnerVolumeSpecName "kube-api-access-7tgvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:08.239159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.239126    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.239159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.239155    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.239159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.239167    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgvq\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-kube-api-access-7tgvq\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.239484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.239181    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a1ff7d-4d50-4776-ac88-70ab88414d78-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.302878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.302825    5424 generic.go:356] "Generic (PLEG): container finished" podID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerID="c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3" exitCode=0
Nov 03 20:33:08.302878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.302862    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerDied","Data":"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"}
Nov 03 20:33:08.303150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.302893    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"
Nov 03 20:33:08.303150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.302922    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp" event={"ID":"c8a1ff7d-4d50-4776-ac88-70ab88414d78","Type":"ContainerDied","Data":"ddbe45d95d275b483260f6935c27d6610c9dde3f3d1cc2be60c6ca75ca140777"}
Nov 03 20:33:08.303150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.302944    5424 scope.go:122] "RemoveContainer" containerID="c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"
Nov 03 20:33:08.314758 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.314732    5424 scope.go:122] "RemoveContainer" containerID="6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"
Nov 03 20:33:08.325426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.325321    5424 scope.go:122] "RemoveContainer" containerID="c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"
Nov 03 20:33:08.325812 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:08.325780    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3\": container with ID starting with c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3 not found: ID does not exist" containerID="c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"
Nov 03 20:33:08.325960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.325816    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3"} err="failed to get container status \"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3\": rpc error: code = NotFound desc = could not find container \"c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3\": container with ID starting with c7b93a250dc4a0a4c1c50c21297344021fc9352517b6190a256aca335e0664e3 not found: ID does not exist"
Nov 03 20:33:08.325960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.325835    5424 scope.go:122] "RemoveContainer" containerID="6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"
Nov 03 20:33:08.326206 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:08.326184    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d\": container with ID starting with 6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d not found: ID does not exist" containerID="6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"
Nov 03 20:33:08.326274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.326219    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d"} err="failed to get container status \"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d\": rpc error: code = NotFound desc = could not find container \"6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d\": container with ID starting with 6f540661f47886f8b8d6e7b20dffeb0c468eaf7986459d7e9a2facbb16ffd93d not found: ID does not exist"
Nov 03 20:33:08.337511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.337476    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:33:08.353170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.353093    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:33:08.366316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.366284    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:33:08.377934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:08.377904    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-17b3d9-openshift-default-87bd856f7-brssp"]
Nov 03 20:33:09.920675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:09.920644    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" path="/var/lib/kubelet/pods/c8a1ff7d-4d50-4776-ac88-70ab88414d78/volumes"
Nov 03 20:33:12.288914 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:12.288864    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:33:12.792394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:12.792354    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"]
Nov 03 20:33:12.802477 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:12.802441    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7e4389-rvh2g"]
Nov 03 20:33:13.919893 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:13.919855    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43359d5a-d8ef-4e98-a400-998191ca09c4" path="/var/lib/kubelet/pods/43359d5a-d8ef-4e98-a400-998191ca09c4/volumes"
Nov 03 20:33:14.813894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:14.813855    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"]
Nov 03 20:33:14.826151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:14.826121    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-e3ba87-6xxxr"]
Nov 03 20:33:15.919875 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:15.919838    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63e36e3-033f-47c4-8762-ffbbe2d0afb7" path="/var/lib/kubelet/pods/b63e36e3-033f-47c4-8762-ffbbe2d0afb7/volumes"
Nov 03 20:33:20.281148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.281095    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"]
Nov 03 20:33:20.282809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282780    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.282809 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282807    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282818    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282826    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282843    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" containerName="revoke-keys"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282851    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" containerName="revoke-keys"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282960    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282968    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a1ff7d-4d50-4776-ac88-70ab88414d78" containerName="istio-proxy"
Nov 03 20:33:20.283028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.282973    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" containerName="revoke-keys"
Nov 03 20:33:20.286876 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.286839    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.300862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.300817    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"]
Nov 03 20:33:20.310459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.310426    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:33:20.310688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.310569    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:33:20.415131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.415093    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpqs\" (UniqueName: \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.415331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.415248    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.415331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.415309    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.516148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.516107    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.516371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.516171    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.516371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.516324    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpqs\" (UniqueName: \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.516953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.516925    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.518880 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.518842    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.536947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.536856    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpqs\" (UniqueName: \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\") pod \"maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.610459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.610421    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:20.915408 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:20.915369    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8f7cbf_ba76_4d64_b24d_32a8a61d849a.slice/crio-40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b WatchSource:0}: Error finding container 40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b: Status 404 returned error can't find the container with id 40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b
Nov 03 20:33:20.916431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:20.916403    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"]
Nov 03 20:33:21.202598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:21.202489    5424 scope.go:122] "RemoveContainer" containerID="3a0239ae42585bba6b5758f403931bda7b7c42edc3ebc4957b06dee8c49ec6b0"
Nov 03 20:33:21.213029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:21.213002    5424 scope.go:122] "RemoveContainer" containerID="1086fc549d13b697714475c4bb0bbd3ed195907c6151b94ea8226cf995af4c4d"
Nov 03 20:33:21.223468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:21.223443    5424 scope.go:122] "RemoveContainer" containerID="a94b660f25291fab9073817b2593a2dd19ea2b068f2bc9eedb51c12b50d4b203"
Nov 03 20:33:21.433400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:21.433363    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l" event={"ID":"df8f7cbf-ba76-4d64-b24d-32a8a61d849a","Type":"ContainerStarted","Data":"40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b"}
Nov 03 20:33:22.426247 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:22.426137    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:33:22.426757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:22.426721    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy" containerID="cri-o://00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783" gracePeriod=30
Nov 03 20:33:22.444791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:22.444749    5424 generic.go:356] "Generic (PLEG): container finished" podID="df8f7cbf-ba76-4d64-b24d-32a8a61d849a" containerID="e04e4c102df5c9ee6e0644d24876c881f349ed52bf0d8f6a3ae2923ba541549d" exitCode=0
Nov 03 20:33:22.445158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:22.444826    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l" event={"ID":"df8f7cbf-ba76-4d64-b24d-32a8a61d849a","Type":"ContainerDied","Data":"e04e4c102df5c9ee6e0644d24876c881f349ed52bf0d8f6a3ae2923ba541549d"}
Nov 03 20:33:23.141366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.141326    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"]
Nov 03 20:33:23.146202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.146182    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.163431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.163398    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"]
Nov 03 20:33:23.169739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.169704    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.169914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.169890    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48flg\" (UniqueName: \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.170096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.170072    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.271786 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.271737    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.272011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.271826    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48flg\" (UniqueName: \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.272011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.271925    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.272432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.272409    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.274508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.274483    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.297513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.297450    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48flg\" (UniqueName: \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\") pod \"maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.360261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.360219    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" containerID="cri-o://e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5" gracePeriod=45
Nov 03 20:33:23.375924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.375887    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main" containerID="cri-o://815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85" gracePeriod=45
Nov 03 20:33:23.461172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.461142    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:23.644892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.644630    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:23.682601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.682499    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\") pod \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") "
Nov 03 20:33:23.682775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.682602    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\") pod \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") "
Nov 03 20:33:23.682843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.682786    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\" (UniqueName: \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\") pod \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\" (UID: \"df8f7cbf-ba76-4d64-b24d-32a8a61d849a\") "
Nov 03 20:33:23.685722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.683311    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca" pod "df8f7cbf-ba76-4d64-b24d-32a8a61d849a" (UID: "df8f7cbf-ba76-4d64-b24d-32a8a61d849a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:23.685722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.683787    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.689954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.688635    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca" pod "df8f7cbf-ba76-4d64-b24d-32a8a61d849a" (UID: "df8f7cbf-ba76-4d64-b24d-32a8a61d849a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:23.689954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.689932    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs" pod "df8f7cbf-ba76-4d64-b24d-32a8a61d849a" (UID: "df8f7cbf-ba76-4d64-b24d-32a8a61d849a"). InnerVolumeSpecName "kube-api-access-8dpqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:23.765091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.765063    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:33:23.780011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.779978    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:33:23.785970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.785934    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\" (UniqueName: \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.786127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.786029    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.789073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.787667    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.789073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.787739    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.789073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.787963    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.790769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.790735    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.790901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.790884    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "kube-api-access-rf8v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:23.793946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.793917    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"]
Nov 03 20:33:23.794062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.788084    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.794062 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794055    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.794183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794128    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\") pod \"d4122f8e-788d-4843-be7e-8bbcf95cb094\" (UID: \"d4122f8e-788d-4843-be7e-8bbcf95cb094\") "
Nov 03 20:33:23.794874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794839    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8v2\" (UniqueName: \"kubernetes.io/projected/d4122f8e-788d-4843-be7e-8bbcf95cb094-kube-api-access-rf8v2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.794990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794877    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.794990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794892    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.794990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794902    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.794990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794913    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.794990 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.794924    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpqs\" (UniqueName: \"kubernetes.io/projected/df8f7cbf-ba76-4d64-b24d-32a8a61d849a-kube-api-access-8dpqs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.797369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.797348    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:23.797801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.797773    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm" pod "d4122f8e-788d-4843-be7e-8bbcf95cb094" (UID: "d4122f8e-788d-4843-be7e-8bbcf95cb094"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.896502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896427    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.896718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896544    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.896718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896599    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.896718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896657    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\" (UniqueName: \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.896974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896793    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.896974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896801    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.896974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.896836    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\") pod \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\" (UID: \"eb80c215-ac2c-40ef-9759-c22a7c6bd717\") "
Nov 03 20:33:23.897175 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897151    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.897444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897388    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.897642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897591    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4122f8e-788d-4843-be7e-8bbcf95cb094-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.897642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897614    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4122f8e-788d-4843-be7e-8bbcf95cb094-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.897642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897627    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.897642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897639    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.897899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.897651    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.899326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.899267    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.899326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.899273    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:23.900567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.900541    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s" pod "eb80c215-ac2c-40ef-9759-c22a7c6bd717" (UID: "eb80c215-ac2c-40ef-9759-c22a7c6bd717"). InnerVolumeSpecName "kube-api-access-79j6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:23.999281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.999234    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79j6s\" (UniqueName: \"kubernetes.io/projected/eb80c215-ac2c-40ef-9759-c22a7c6bd717-kube-api-access-79j6s\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.999281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.999280    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb80c215-ac2c-40ef-9759-c22a7c6bd717-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.999281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:23.999295    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb80c215-ac2c-40ef-9759-c22a7c6bd717-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.460687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.460646    5424 generic.go:356] "Generic (PLEG): container finished" podID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerID="815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85" exitCode=0
Nov 03 20:33:24.460901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.460720    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"
Nov 03 20:33:24.460901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.460731    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" event={"ID":"eb80c215-ac2c-40ef-9759-c22a7c6bd717","Type":"ContainerDied","Data":"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"}
Nov 03 20:33:24.460901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.460784    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx" event={"ID":"eb80c215-ac2c-40ef-9759-c22a7c6bd717","Type":"ContainerDied","Data":"c693a9d03ff386e77f944e6c3288a6329fd89ca5941846599a21325fa433d0b4"}
Nov 03 20:33:24.460901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.460805    5424 scope.go:122] "RemoveContainer" containerID="815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"
Nov 03 20:33:24.462675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.462642    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs" event={"ID":"f3e3ee72-11e5-470c-b9f0-8ecb392b5663","Type":"ContainerStarted","Data":"0d852df7ec6571574b2ecd32fa8c853a39546f2a79b6773f6ada7f5797e5eb3e"}
Nov 03 20:33:24.463036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.462704    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs" event={"ID":"f3e3ee72-11e5-470c-b9f0-8ecb392b5663","Type":"ContainerStarted","Data":"596aedf9020a493b2d291dabbf4ae0c4daf8e88bafe69eabc9d023d2cd0beb45"}
Nov 03 20:33:24.464249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.464232    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"
Nov 03 20:33:24.464369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.464270    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l" event={"ID":"df8f7cbf-ba76-4d64-b24d-32a8a61d849a","Type":"ContainerDied","Data":"40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b"}
Nov 03 20:33:24.464369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.464310    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40c82b6947af286f5db5ecc2a250e52fcf8b5c09f6b7e53319e9130dc3a7d16b"
Nov 03 20:33:24.465703 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.465682    5424 generic.go:356] "Generic (PLEG): container finished" podID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerID="e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5" exitCode=0
Nov 03 20:33:24.465826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.465774    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" event={"ID":"d4122f8e-788d-4843-be7e-8bbcf95cb094","Type":"ContainerDied","Data":"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"}
Nov 03 20:33:24.465826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.465792    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"
Nov 03 20:33:24.465826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.465800    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" event={"ID":"d4122f8e-788d-4843-be7e-8bbcf95cb094","Type":"ContainerDied","Data":"53a75922cf2705017fcc02ee7c8fccd7ed7896cc6dfeb5c80c5a5aef48867b00"}
Nov 03 20:33:24.472342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.472323    5424 scope.go:122] "RemoveContainer" containerID="815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"
Nov 03 20:33:24.472654 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:24.472630    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85\": container with ID starting with 815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85 not found: ID does not exist" containerID="815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"
Nov 03 20:33:24.472775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.472662    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85"} err="failed to get container status \"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85\": rpc error: code = NotFound desc = could not find container \"815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85\": container with ID starting with 815a09fc529781b55a03596c130691a1480d9bfe7d294e9a3d3a8c8190e68e85 not found: ID does not exist"
Nov 03 20:33:24.472775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.472686    5424 scope.go:122] "RemoveContainer" containerID="e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"
Nov 03 20:33:24.482318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.482298    5424 scope.go:122] "RemoveContainer" containerID="e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"
Nov 03 20:33:24.482610 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:24.482589    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5\": container with ID starting with e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5 not found: ID does not exist" containerID="e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"
Nov 03 20:33:24.482688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.482620    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5"} err="failed to get container status \"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5\": rpc error: code = NotFound desc = could not find container \"e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5\": container with ID starting with e5f4d3a106acee40594c410ed3ba16e216dcafced2ad086b6e256aed1564a7c5 not found: ID does not exist"
Nov 03 20:33:24.528419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.528386    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:33:24.540428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.540358    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8"]
Nov 03 20:33:24.575306 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.575252    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs" podStartSLOduration=1.5752382150000002 podStartE2EDuration="1.575238215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:24.574245959 +0000 UTC m=+1746.783261957" watchObservedRunningTime="2026-11-03 20:33:24.575238215 +0000 UTC m=+1746.784254211"
Nov 03 20:33:24.610892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.610844    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:33:24.613924 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.613896    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/simulator-w6-9e1bcb-kserve-56f8d9599c-2gwhx"]
Nov 03 20:33:24.701782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:24.701721    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-w6-9e1bcb-kserve-686746b8bc-kxkl8" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:25.025303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.025268    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:33:25.029810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.029786    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:33:25.050721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.050692    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:33:25.477545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.477506    5424 generic.go:356] "Generic (PLEG): container finished" podID="f3e3ee72-11e5-470c-b9f0-8ecb392b5663" containerID="0d852df7ec6571574b2ecd32fa8c853a39546f2a79b6773f6ada7f5797e5eb3e" exitCode=0
Nov 03 20:33:25.477982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.477563    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs" event={"ID":"f3e3ee72-11e5-470c-b9f0-8ecb392b5663","Type":"ContainerDied","Data":"0d852df7ec6571574b2ecd32fa8c853a39546f2a79b6773f6ada7f5797e5eb3e"}
Nov 03 20:33:25.920907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.920875    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" path="/var/lib/kubelet/pods/d4122f8e-788d-4843-be7e-8bbcf95cb094/volumes"
Nov 03 20:33:25.921335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:25.921321    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" path="/var/lib/kubelet/pods/eb80c215-ac2c-40ef-9759-c22a7c6bd717/volumes"
Nov 03 20:33:26.636275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.636248    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:26.738018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.737931    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\") pod \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") "
Nov 03 20:33:26.738166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.738124    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\" (UniqueName: \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\") pod \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") "
Nov 03 20:33:26.738217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.738206    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\") pod \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\" (UID: \"f3e3ee72-11e5-470c-b9f0-8ecb392b5663\") "
Nov 03 20:33:26.738327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.738297    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca" pod "f3e3ee72-11e5-470c-b9f0-8ecb392b5663" (UID: "f3e3ee72-11e5-470c-b9f0-8ecb392b5663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:26.738630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.738607    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:26.740303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.740280    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca" pod "f3e3ee72-11e5-470c-b9f0-8ecb392b5663" (UID: "f3e3ee72-11e5-470c-b9f0-8ecb392b5663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:26.740389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.740366    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg" pod "f3e3ee72-11e5-470c-b9f0-8ecb392b5663" (UID: "f3e3ee72-11e5-470c-b9f0-8ecb392b5663"). InnerVolumeSpecName "kube-api-access-48flg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:26.839650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.839614    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48flg\" (UniqueName: \"kubernetes.io/projected/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-kube-api-access-48flg\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:26.839650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:26.839644    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e3ee72-11e5-470c-b9f0-8ecb392b5663-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.493956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.493921    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"
Nov 03 20:33:27.494139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.493921    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs" event={"ID":"f3e3ee72-11e5-470c-b9f0-8ecb392b5663","Type":"ContainerDied","Data":"596aedf9020a493b2d291dabbf4ae0c4daf8e88bafe69eabc9d023d2cd0beb45"}
Nov 03 20:33:27.494139 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.494071    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596aedf9020a493b2d291dabbf4ae0c4daf8e88bafe69eabc9d023d2cd0beb45"
Nov 03 20:33:27.726918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.726888    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:33:27.855993 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.855959    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856084    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856129    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856164    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856198    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856267    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856345    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:27.856453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856447    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856466    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:27.856666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856513    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856564    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\") pod \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\" (UID: \"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d\") "
Nov 03 20:33:27.856818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856691    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:27.856996 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.856971    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:27.858017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.857168    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.858017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.857192    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.858017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.857207    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.858017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.857223    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.858017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.857507    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:27.861338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.860175    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "kube-api-access-59kh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:27.861338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.860523    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:27.861776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.861463    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:27.868582 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.866075    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:33:27.871683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.871654    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo" pod "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" (UID: "fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:27.884085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.884056    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:33:27.891771 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.891735    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:33:27.959038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.958948    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.959038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.958978    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59kh2\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-kube-api-access-59kh2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.959038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.958989    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.959038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.958998    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:27.959038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:27.959006    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.501928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.501887    5424 generic.go:356] "Generic (PLEG): container finished" podID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerID="00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783" exitCode=0
Nov 03 20:33:28.502115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.501935    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"
Nov 03 20:33:28.502115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.501977    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" event={"ID":"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d","Type":"ContainerDied","Data":"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"}
Nov 03 20:33:28.502115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.502035    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6" event={"ID":"fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d","Type":"ContainerDied","Data":"500b22e963fca0a0107b86dbead0e5f1d64d382dfee95d0cd07167f645620c87"}
Nov 03 20:33:28.502115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.502064    5424 scope.go:122] "RemoveContainer" containerID="00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"
Nov 03 20:33:28.514740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.514717    5424 scope.go:122] "RemoveContainer" containerID="00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"
Nov 03 20:33:28.515086 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:28.515062    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783\": container with ID starting with 00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783 not found: ID does not exist" containerID="00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"
Nov 03 20:33:28.515195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.515096    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783"} err="failed to get container status \"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783\": rpc error: code = NotFound desc = could not find container \"00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783\": container with ID starting with 00c3b042c931e0434308cd8b9a8f9906bbd864586132573e187047f76fdd5783 not found: ID does not exist"
Nov 03 20:33:28.562611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.562577    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:33:28.575821 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:28.575790    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-wwzd6"]
Nov 03 20:33:29.920253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:29.920216    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" path="/var/lib/kubelet/pods/fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d/volumes"
Nov 03 20:33:30.561004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.560964    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"]
Nov 03 20:33:30.561503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561489    5424 cpu_manager.go:414] "containerMap: removing container" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main"
Nov 03 20:33:30.561503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561503    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561543    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561552    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561561    5424 cpu_manager.go:414] "containerMap: removing container" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561566    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561574    5424 cpu_manager.go:414] "containerMap: removing container" podUID="df8f7cbf-ba76-4d64-b24d-32a8a61d849a" containerName="revoke-keys"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561578    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8f7cbf-ba76-4d64-b24d-32a8a61d849a" containerName="revoke-keys"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561601    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f3e3ee72-11e5-470c-b9f0-8ecb392b5663" containerName="revoke-keys"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561608    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e3ee72-11e5-470c-b9f0-8ecb392b5663" containerName="revoke-keys"
Nov 03 20:33:30.561691 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561696    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4122f8e-788d-4843-be7e-8bbcf95cb094" containerName="main"
Nov 03 20:33:30.561999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561712    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4c1c45-82d9-4f8d-b5d9-5be3cc09c03d" containerName="istio-proxy"
Nov 03 20:33:30.561999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561720    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8f7cbf-ba76-4d64-b24d-32a8a61d849a" containerName="revoke-keys"
Nov 03 20:33:30.561999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561729    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e3ee72-11e5-470c-b9f0-8ecb392b5663" containerName="revoke-keys"
Nov 03 20:33:30.561999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.561737    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb80c215-ac2c-40ef-9759-c22a7c6bd717" containerName="main"
Nov 03 20:33:30.566712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.566691    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.582286 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.582246    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"]
Nov 03 20:33:30.590200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.590176    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:33:30.590379 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.590356    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:33:30.590574 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.590548    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"]
Nov 03 20:33:30.595431 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.595411    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.613864 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.613834    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"]
Nov 03 20:33:30.689366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689332    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.689590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689380    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkzf\" (UniqueName: \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.689590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689491    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.689590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689510    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.689789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689644    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gld4\" (UniqueName: \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.689845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.689800    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.790943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.790904    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.790943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.790945    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.791190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.791016    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gld4\" (UniqueName: \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.791190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.791129    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.791270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.791208    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.791270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.791245    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkzf\" (UniqueName: \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.791795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.791770    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.792032 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.792007    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.793723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.793691    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.793773 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.793734    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.810980 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.810951    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkzf\" (UniqueName: \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\") pod \"maas-api-revoke-sub-659534aa-xnqhs\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:30.811171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.810951    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gld4\" (UniqueName: \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\") pod \"maas-api-revoke-sub-70fea5d1-w72mg\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.881633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.881596    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:30.914351 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:30.914315    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:31.209185 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:31.209147    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdcd131_c5f8_4d9b_aaa1_7e35182fb01e.slice/crio-a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39 WatchSource:0}: Error finding container a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39: Status 404 returned error can't find the container with id a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39
Nov 03 20:33:31.212971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:31.212945    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"]
Nov 03 20:33:31.245735 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:31.245691    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb219ae6c_acf8_4bde_a986_fef60824fada.slice/crio-b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974 WatchSource:0}: Error finding container b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974: Status 404 returned error can't find the container with id b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974
Nov 03 20:33:31.245861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:31.245798    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"]
Nov 03 20:33:31.538487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:31.538457    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg" event={"ID":"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e","Type":"ContainerStarted","Data":"a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39"}
Nov 03 20:33:31.540376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:31.540349    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs" event={"ID":"b219ae6c-acf8-4bde-a986-fef60824fada","Type":"ContainerStarted","Data":"b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974"}
Nov 03 20:33:31.585848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:31.585798    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs" podStartSLOduration=1.585784051 podStartE2EDuration="1.585784051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:31.581338994 +0000 UTC m=+1753.790354988" watchObservedRunningTime="2026-11-03 20:33:31.585784051 +0000 UTC m=+1753.794800048"
Nov 03 20:33:32.552468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:32.552368    5424 generic.go:356] "Generic (PLEG): container finished" podID="4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" containerID="503b2da7bd96f94f46a5e39d50f02226807969d99f521244d6073877ae432b08" exitCode=0
Nov 03 20:33:32.552468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:32.552447    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg" event={"ID":"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e","Type":"ContainerDied","Data":"503b2da7bd96f94f46a5e39d50f02226807969d99f521244d6073877ae432b08"}
Nov 03 20:33:32.554581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:32.554521    5424 generic.go:356] "Generic (PLEG): container finished" podID="b219ae6c-acf8-4bde-a986-fef60824fada" containerID="a4d9f8aba785fc8113a76423e2cc00a1351844b6639a9ed31864967a29d0e8b8" exitCode=0
Nov 03 20:33:32.554726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:32.554559    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs" event={"ID":"b219ae6c-acf8-4bde-a986-fef60824fada","Type":"ContainerDied","Data":"a4d9f8aba785fc8113a76423e2cc00a1351844b6639a9ed31864967a29d0e8b8"}
Nov 03 20:33:33.815784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.815748    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:33.868558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.867847    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:33.942853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.942812    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\") pod \"b219ae6c-acf8-4bde-a986-fef60824fada\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") "
Nov 03 20:33:33.943023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.942942    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\" (UniqueName: \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\") pod \"b219ae6c-acf8-4bde-a986-fef60824fada\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") "
Nov 03 20:33:33.943104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.943065    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\") pod \"b219ae6c-acf8-4bde-a986-fef60824fada\" (UID: \"b219ae6c-acf8-4bde-a986-fef60824fada\") "
Nov 03 20:33:33.943242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.943216    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca" pod "b219ae6c-acf8-4bde-a986-fef60824fada" (UID: "b219ae6c-acf8-4bde-a986-fef60824fada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:33.943651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.943626    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:33.945511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.945480    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf" pod "b219ae6c-acf8-4bde-a986-fef60824fada" (UID: "b219ae6c-acf8-4bde-a986-fef60824fada"). InnerVolumeSpecName "kube-api-access-fjkzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:33.945802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:33.945772    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca" pod "b219ae6c-acf8-4bde-a986-fef60824fada" (UID: "b219ae6c-acf8-4bde-a986-fef60824fada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:34.044402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.044369    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\" (UniqueName: \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\") pod \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") "
Nov 03 20:33:34.044588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.044504    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\") pod \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") "
Nov 03 20:33:34.044664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.044610    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\") pod \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\" (UID: \"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e\") "
Nov 03 20:33:34.044966 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.044939    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca" pod "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" (UID: "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:34.045153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.045137    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b219ae6c-acf8-4bde-a986-fef60824fada-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:34.045401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.045155    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkzf\" (UniqueName: \"kubernetes.io/projected/b219ae6c-acf8-4bde-a986-fef60824fada-kube-api-access-fjkzf\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:34.045401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.045169    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:34.046818 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.046793    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca" pod "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" (UID: "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:34.046934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.046902    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4" pod "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" (UID: "4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e"). InnerVolumeSpecName "kube-api-access-9gld4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:34.146845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.146764    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gld4\" (UniqueName: \"kubernetes.io/projected/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-kube-api-access-9gld4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:34.146845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.146803    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:34.578848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.578814    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"
Nov 03 20:33:34.579123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.578847    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg" event={"ID":"4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e","Type":"ContainerDied","Data":"a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39"}
Nov 03 20:33:34.579123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.578913    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ee8cab6ab75fccde2939d25277e25cb5904206d0cdfb17d33f6d8dc321df39"
Nov 03 20:33:34.580654 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.580627    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs" event={"ID":"b219ae6c-acf8-4bde-a986-fef60824fada","Type":"ContainerDied","Data":"b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974"}
Nov 03 20:33:34.580813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.580676    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b86a44ced967a840ea6864f2b01d22ccc9b66a0b219c50ce65844cd5ddaf9974"
Nov 03 20:33:34.580813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:34.580640    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"
Nov 03 20:33:38.031293 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:38.031261    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"]
Nov 03 20:33:38.043661 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:38.043626    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b84599c-nl7x4"]
Nov 03 20:33:39.828202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:39.828163    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"]
Nov 03 20:33:39.840476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:39.840439    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d05604-4dbcq"]
Nov 03 20:33:39.923776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:39.923728    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58333060-939f-476b-b071-53af2ab33053" path="/var/lib/kubelet/pods/58333060-939f-476b-b071-53af2ab33053/volumes"
Nov 03 20:33:39.924448 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:39.924423    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faf616e-5358-4364-8c68-97d8363b2e96" path="/var/lib/kubelet/pods/5faf616e-5358-4364-8c68-97d8363b2e96/volumes"
Nov 03 20:33:40.047890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.047846    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main" containerID="cri-o://34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929" gracePeriod=45
Nov 03 20:33:40.051692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.051648    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main" containerID="cri-o://4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf" gracePeriod=45
Nov 03 20:33:40.071096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.071052    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main" containerID="cri-o://b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36" gracePeriod=45
Nov 03 20:33:40.392613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.389953    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:33:40.414381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.414341    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:33:40.435734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.435707    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:33:40.437000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.436975    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437017    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\" (UniqueName: \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437056    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.437152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437085    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.437152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437115    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437138    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\" (UniqueName: \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437175    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.437426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437212    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.437426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437268    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.437426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437297    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437426 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437344    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\") pod \"7acb85c4-bd6f-4041-8a15-ae289c527c26\" (UID: \"7acb85c4-bd6f-4041-8a15-ae289c527c26\") "
Nov 03 20:33:40.437726 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.437447    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\" (UniqueName: \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\") pod \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\" (UID: \"1e3429f7-df22-46d3-917b-d71fa0cf5ee6\") "
Nov 03 20:33:40.438458 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.438429    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.438458 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.438436    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.438782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.438757    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.439021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.438999    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.439481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.439458    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.439590 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.439568    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.442327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.442216    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.446294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.446050    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.446294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.446154    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "kube-api-access-q28fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:40.446983 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.446942    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs" pod "7acb85c4-bd6f-4041-8a15-ae289c527c26" (UID: "7acb85c4-bd6f-4041-8a15-ae289c527c26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:40.447216 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.447183    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:40.449082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.449059    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7" pod "1e3429f7-df22-46d3-917b-d71fa0cf5ee6" (UID: "1e3429f7-df22-46d3-917b-d71fa0cf5ee6"). InnerVolumeSpecName "kube-api-access-rl6g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:40.538742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.538710    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.538989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.538803    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.538989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.538920    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\" (UniqueName: \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.538989 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.538972    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.539160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.539122    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.539213 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.539182    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\") pod \"d7d0a6a1-5392-493e-b770-d9a6252975db\" (UID: \"d7d0a6a1-5392-493e-b770-d9a6252975db\") "
Nov 03 20:33:40.539902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.539588    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.539902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.539870    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.539902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.539888    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540015    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540035    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540049    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540061    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540076    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6g7\" (UniqueName: \"kubernetes.io/projected/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-kube-api-access-rl6g7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540089 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540089    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540101    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28fh\" (UniqueName: \"kubernetes.io/projected/7acb85c4-bd6f-4041-8a15-ae289c527c26-kube-api-access-q28fh\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540113    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540126    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540138    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540150    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7acb85c4-bd6f-4041-8a15-ae289c527c26-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540162    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acb85c4-bd6f-4041-8a15-ae289c527c26-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540174    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540186    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.540261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.540197    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3429f7-df22-46d3-917b-d71fa0cf5ee6-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.542346 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.542302    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "kube-api-access-5728t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:40.542601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.542575    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:40.542742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.542718    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm" pod "d7d0a6a1-5392-493e-b770-d9a6252975db" (UID: "d7d0a6a1-5392-493e-b770-d9a6252975db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.634675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.634624    5424 generic.go:356] "Generic (PLEG): container finished" podID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerID="34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929" exitCode=0
Nov 03 20:33:40.634862 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.634825    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" event={"ID":"1e3429f7-df22-46d3-917b-d71fa0cf5ee6","Type":"ContainerDied","Data":"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"}
Nov 03 20:33:40.634962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.634862    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7" event={"ID":"1e3429f7-df22-46d3-917b-d71fa0cf5ee6","Type":"ContainerDied","Data":"144a41be5019811f89e046effa7a042d38db2718bd6bfbc0c4f6cc506398b75d"}
Nov 03 20:33:40.634962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.634884    5424 scope.go:122] "RemoveContainer" containerID="34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"
Nov 03 20:33:40.642553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.635055    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"
Nov 03 20:33:40.643699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.643555    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d0a6a1-5392-493e-b770-d9a6252975db-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.643699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.643588    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5728t\" (UniqueName: \"kubernetes.io/projected/d7d0a6a1-5392-493e-b770-d9a6252975db-kube-api-access-5728t\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.643699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.643603    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d0a6a1-5392-493e-b770-d9a6252975db-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.644928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.644895    5424 generic.go:356] "Generic (PLEG): container finished" podID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerID="4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf" exitCode=0
Nov 03 20:33:40.645055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.644938    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"
Nov 03 20:33:40.645055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.644992    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" event={"ID":"7acb85c4-bd6f-4041-8a15-ae289c527c26","Type":"ContainerDied","Data":"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"}
Nov 03 20:33:40.645055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.645032    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n" event={"ID":"7acb85c4-bd6f-4041-8a15-ae289c527c26","Type":"ContainerDied","Data":"391ce8f26510b913bdb23d24b4bf849f459bf31c206604a2f0561e7e6650215c"}
Nov 03 20:33:40.647361 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.647333    5424 generic.go:356] "Generic (PLEG): container finished" podID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerID="b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36" exitCode=0
Nov 03 20:33:40.647472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.647395    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"
Nov 03 20:33:40.647472 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.647437    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" event={"ID":"d7d0a6a1-5392-493e-b770-d9a6252975db","Type":"ContainerDied","Data":"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"}
Nov 03 20:33:40.647629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.647480    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt" event={"ID":"d7d0a6a1-5392-493e-b770-d9a6252975db","Type":"ContainerDied","Data":"0c7b5bda9271de5ec3fe5014fcdcf730ce7751ad04734f71f9733f6273e0328c"}
Nov 03 20:33:40.659140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.656914    5424 scope.go:122] "RemoveContainer" containerID="34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"
Nov 03 20:33:40.659140 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:40.657967    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929\": container with ID starting with 34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929 not found: ID does not exist" containerID="34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"
Nov 03 20:33:40.659140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.658009    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929"} err="failed to get container status \"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929\": rpc error: code = NotFound desc = could not find container \"34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929\": container with ID starting with 34fc1a340f89b48d071d33c2d5c6f0d94513ea179a1b4e1b0000139401faf929 not found: ID does not exist"
Nov 03 20:33:40.659140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.658035    5424 scope.go:122] "RemoveContainer" containerID="4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"
Nov 03 20:33:40.676277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.676254    5424 scope.go:122] "RemoveContainer" containerID="4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"
Nov 03 20:33:40.676608 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:40.676586    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf\": container with ID starting with 4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf not found: ID does not exist" containerID="4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"
Nov 03 20:33:40.676648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.676622    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf"} err="failed to get container status \"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf\": rpc error: code = NotFound desc = could not find container \"4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf\": container with ID starting with 4ffed3ba2cb9700713d07ea6700a8057e21532bca928198042869fb2fcbbbfdf not found: ID does not exist"
Nov 03 20:33:40.676648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.676639    5424 scope.go:122] "RemoveContainer" containerID="b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"
Nov 03 20:33:40.687449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.687393    5424 scope.go:122] "RemoveContainer" containerID="b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"
Nov 03 20:33:40.687737 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:40.687710    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36\": container with ID starting with b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36 not found: ID does not exist" containerID="b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"
Nov 03 20:33:40.687841 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.687745    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36"} err="failed to get container status \"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36\": rpc error: code = NotFound desc = could not find container \"b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36\": container with ID starting with b7e34eb44c1d2c7c1c3c1813c01ed3f84c7c38b2677dd222de89239211d78a36 not found: ID does not exist"
Nov 03 20:33:40.697649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.697617    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:33:40.701672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.701647    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-2-w2-9973d9-kserve-68867b96c-skpv7"]
Nov 03 20:33:40.763202 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.763165    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:33:40.772962 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.772927    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/simulator-w2-9973d9-kserve-86769df68-kjbbt"]
Nov 03 20:33:40.813239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.813199    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:33:40.817384 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:40.817352    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/distinct-w2-9973d9-kserve-dffd5b4bd-vk28n"]
Nov 03 20:33:41.406457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406377    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"]
Nov 03 20:33:41.406934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406919    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main"
Nov 03 20:33:41.406974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406934    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main"
Nov 03 20:33:41.406974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406948    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main"
Nov 03 20:33:41.406974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406953    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406974    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406981    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406989    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b219ae6c-acf8-4bde-a986-fef60824fada" containerName="revoke-keys"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.406994    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b219ae6c-acf8-4bde-a986-fef60824fada" containerName="revoke-keys"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407007    5424 cpu_manager.go:414] "containerMap: removing container" podUID="4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" containerName="revoke-keys"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407011    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" containerName="revoke-keys"
Nov 03 20:33:41.407078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407079    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" containerName="main"
Nov 03 20:33:41.407283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407092    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" containerName="main"
Nov 03 20:33:41.407283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407098    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b219ae6c-acf8-4bde-a986-fef60824fada" containerName="revoke-keys"
Nov 03 20:33:41.407283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407107    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" containerName="revoke-keys"
Nov 03 20:33:41.407283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.407114    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" containerName="main"
Nov 03 20:33:41.411860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.411835    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.426331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.426293    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"]
Nov 03 20:33:41.431046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.431007    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"]
Nov 03 20:33:41.431199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.431099    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.435485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.435458    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:33:41.436674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.436587    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:33:41.451107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451065    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.451352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451332    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9hp\" (UniqueName: \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.451506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451487    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.451667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451644    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"]
Nov 03 20:33:41.451858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451841    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.451930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451910    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.451988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.451958    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62ds\" (UniqueName: \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.553423 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553387    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62ds\" (UniqueName: \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.553667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553457    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.553667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553555    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9hp\" (UniqueName: \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.553667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553598    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.553895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553670    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.553895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.553694    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.554316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.554282    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.554578 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.554324    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.556299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.556271    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.556404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.556369    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.573300 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.573267    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9hp\" (UniqueName: \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\") pod \"maas-api-revoke-sub-cce99c8d-rbz76\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.573677 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.573654    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62ds\" (UniqueName: \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\") pod \"maas-api-revoke-sub-fab1bb26-4t9h7\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.727934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.727843    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:41.752586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.752549    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:41.923027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.922989    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3429f7-df22-46d3-917b-d71fa0cf5ee6" path="/var/lib/kubelet/pods/1e3429f7-df22-46d3-917b-d71fa0cf5ee6/volumes"
Nov 03 20:33:41.923572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.923551    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acb85c4-bd6f-4041-8a15-ae289c527c26" path="/var/lib/kubelet/pods/7acb85c4-bd6f-4041-8a15-ae289c527c26/volumes"
Nov 03 20:33:41.924037 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:41.924018    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d0a6a1-5392-493e-b770-d9a6252975db" path="/var/lib/kubelet/pods/d7d0a6a1-5392-493e-b770-d9a6252975db/volumes"
Nov 03 20:33:42.057433 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:42.057395    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772286d7_8bcd_4fb6_a143_ade7c5d4488c.slice/crio-a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064 WatchSource:0}: Error finding container a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064: Status 404 returned error can't find the container with id a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064
Nov 03 20:33:42.057899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.057856    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"]
Nov 03 20:33:42.096800 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.096775    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"]
Nov 03 20:33:42.100873 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:42.100843    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267a1961_2acb_435e_b8cb_e2937570c03c.slice/crio-a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431 WatchSource:0}: Error finding container a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431: Status 404 returned error can't find the container with id a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431
Nov 03 20:33:42.286257 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:42.286222    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:33:42.666887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.666776    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76" event={"ID":"772286d7-8bcd-4fb6-a143-ade7c5d4488c","Type":"ContainerStarted","Data":"d325f76478f072d975f241ef29272d311bfea8a1a755b4d9fcf85038187fb49b"}
Nov 03 20:33:42.666887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.666844    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76" event={"ID":"772286d7-8bcd-4fb6-a143-ade7c5d4488c","Type":"ContainerStarted","Data":"a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064"}
Nov 03 20:33:42.668675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.668642    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7" event={"ID":"267a1961-2acb-435e-b8cb-e2937570c03c","Type":"ContainerStarted","Data":"b585238f5aa02422a0c69efafd67d65476fd0818b35d34d8bd70e214431eed9f"}
Nov 03 20:33:42.668798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.668701    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7" event={"ID":"267a1961-2acb-435e-b8cb-e2937570c03c","Type":"ContainerStarted","Data":"a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431"}
Nov 03 20:33:42.706973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.706915    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76" podStartSLOduration=1.706901889 podStartE2EDuration="1.706901889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:42.705860584 +0000 UTC m=+1764.914876581" watchObservedRunningTime="2026-11-03 20:33:42.706901889 +0000 UTC m=+1764.915917886"
Nov 03 20:33:42.748884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.748827    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7" podStartSLOduration=1.7488138659999999 podStartE2EDuration="1.748813866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:42.745630866 +0000 UTC m=+1764.954646865" watchObservedRunningTime="2026-11-03 20:33:42.748813866 +0000 UTC m=+1764.957829875"
Nov 03 20:33:42.892515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.892454    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" containerID="cri-o://542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1" gracePeriod=45
Nov 03 20:33:42.916090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.916019    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" containerID="cri-o://1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d" gracePeriod=45
Nov 03 20:33:42.916090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:42.916066    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main" containerID="cri-o://16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387" gracePeriod=45
Nov 03 20:33:43.248751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.248721    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:33:43.272890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.272811    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:33:43.275770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.275739    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\" (UniqueName: \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.275949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.275875    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.275949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.275910    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.275949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.275941    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.276143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.276039    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.276200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.276163    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\") pod \"245b12cc-7342-4444-8e86-d85ed0e34f5a\" (UID: \"245b12cc-7342-4444-8e86-d85ed0e34f5a\") "
Nov 03 20:33:43.276449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.276296    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:33:43.277260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.276786    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.277260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.276931    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.277513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.277478    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.279558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.279495    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.279673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.279648    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "kube-api-access-6qnwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:43.279873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.279852    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs" pod "245b12cc-7342-4444-8e86-d85ed0e34f5a" (UID: "245b12cc-7342-4444-8e86-d85ed0e34f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:43.378011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.377960    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\" (UniqueName: \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378057    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378212    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378268    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.378368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378312    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378344    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.378502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378353    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.378502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378447    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378480    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.378720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378578    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.378720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378597    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\" (UniqueName: \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.378720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378693    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\") pod \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\" (UID: \"f42f85c3-00f2-4335-9556-dc8882d2f4fd\") "
Nov 03 20:33:43.378884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378741    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.378884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378766    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.378884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.378810    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\") pod \"a18071f0-b070-4d57-aa55-a6ae470f34b5\" (UID: \"a18071f0-b070-4d57-aa55-a6ae470f34b5\") "
Nov 03 20:33:43.379350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.379323    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qnwk\" (UniqueName: \"kubernetes.io/projected/245b12cc-7342-4444-8e86-d85ed0e34f5a-kube-api-access-6qnwk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.379610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.379352    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.379610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.379384    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.379796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.379772    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.379397    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380274    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380290    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380307    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380320    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/245b12cc-7342-4444-8e86-d85ed0e34f5a-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380332    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245b12cc-7342-4444-8e86-d85ed0e34f5a-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380651    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.380863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.380711    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.382453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.382413    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.382453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.382418    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "kube-api-access-5p62s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:43.383729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.383707    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "kube-api-access-x5x78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:43.383947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.383933    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:43.384039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.384026    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs" pod "a18071f0-b070-4d57-aa55-a6ae470f34b5" (UID: "a18071f0-b070-4d57-aa55-a6ae470f34b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:43.384325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.384311    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm" pod "f42f85c3-00f2-4335-9556-dc8882d2f4fd" (UID: "f42f85c3-00f2-4335-9556-dc8882d2f4fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:43.481895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481866    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.481895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481897    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p62s\" (UniqueName: \"kubernetes.io/projected/f42f85c3-00f2-4335-9556-dc8882d2f4fd-kube-api-access-5p62s\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481908    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481918    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481927    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481935    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a18071f0-b070-4d57-aa55-a6ae470f34b5-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481942    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5x78\" (UniqueName: \"kubernetes.io/projected/a18071f0-b070-4d57-aa55-a6ae470f34b5-kube-api-access-x5x78\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481949    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f42f85c3-00f2-4335-9556-dc8882d2f4fd-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.482082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.481960    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a18071f0-b070-4d57-aa55-a6ae470f34b5-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:43.677944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.677908    5424 generic.go:356] "Generic (PLEG): container finished" podID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerID="542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1" exitCode=0
Nov 03 20:33:43.678436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.677967    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"
Nov 03 20:33:43.678436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.678010    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" event={"ID":"a18071f0-b070-4d57-aa55-a6ae470f34b5","Type":"ContainerDied","Data":"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"}
Nov 03 20:33:43.678436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.678059    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" event={"ID":"a18071f0-b070-4d57-aa55-a6ae470f34b5","Type":"ContainerDied","Data":"56ece64f324234528521fedeb8e03c85d42789ca4487c4770d2c10ca55af97ea"}
Nov 03 20:33:43.678436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.678082    5424 scope.go:122] "RemoveContainer" containerID="542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"
Nov 03 20:33:43.680126 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.680099    5424 generic.go:356] "Generic (PLEG): container finished" podID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerID="1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d" exitCode=0
Nov 03 20:33:43.680270 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.680255    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"
Nov 03 20:33:43.680339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.680296    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" event={"ID":"245b12cc-7342-4444-8e86-d85ed0e34f5a","Type":"ContainerDied","Data":"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"}
Nov 03 20:33:43.680417 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.680354    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" event={"ID":"245b12cc-7342-4444-8e86-d85ed0e34f5a","Type":"ContainerDied","Data":"382d92ec374a639ed382d794d1150150536a156eff23108cc2061d2cf18d977a"}
Nov 03 20:33:43.682006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.681980    5424 generic.go:356] "Generic (PLEG): container finished" podID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerID="16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387" exitCode=0
Nov 03 20:33:43.682134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.682037    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"
Nov 03 20:33:43.682134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.682067    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" event={"ID":"f42f85c3-00f2-4335-9556-dc8882d2f4fd","Type":"ContainerDied","Data":"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"}
Nov 03 20:33:43.682134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.682098    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4" event={"ID":"f42f85c3-00f2-4335-9556-dc8882d2f4fd","Type":"ContainerDied","Data":"adcbd849fde9483d1bc2f666eed46835e6b390086c63836d9da36acdafdcba67"}
Nov 03 20:33:43.684662 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.684636    5424 generic.go:356] "Generic (PLEG): container finished" podID="772286d7-8bcd-4fb6-a143-ade7c5d4488c" containerID="d325f76478f072d975f241ef29272d311bfea8a1a755b4d9fcf85038187fb49b" exitCode=0
Nov 03 20:33:43.684796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.684721    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76" event={"ID":"772286d7-8bcd-4fb6-a143-ade7c5d4488c","Type":"ContainerDied","Data":"d325f76478f072d975f241ef29272d311bfea8a1a755b4d9fcf85038187fb49b"}
Nov 03 20:33:43.686462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.686434    5424 generic.go:356] "Generic (PLEG): container finished" podID="267a1961-2acb-435e-b8cb-e2937570c03c" containerID="b585238f5aa02422a0c69efafd67d65476fd0818b35d34d8bd70e214431eed9f" exitCode=0
Nov 03 20:33:43.686599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.686480    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7" event={"ID":"267a1961-2acb-435e-b8cb-e2937570c03c","Type":"ContainerDied","Data":"b585238f5aa02422a0c69efafd67d65476fd0818b35d34d8bd70e214431eed9f"}
Nov 03 20:33:43.692326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.692208    5424 scope.go:122] "RemoveContainer" containerID="542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"
Nov 03 20:33:43.692652 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:43.692626    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1\": container with ID starting with 542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1 not found: ID does not exist" containerID="542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"
Nov 03 20:33:43.692754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.692661    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1"} err="failed to get container status \"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1\": rpc error: code = NotFound desc = could not find container \"542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1\": container with ID starting with 542bc9af2488612e1c49419dde86a8be68b52688a889a8da7042ac04899a99f1 not found: ID does not exist"
Nov 03 20:33:43.692754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.692679    5424 scope.go:122] "RemoveContainer" containerID="1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"
Nov 03 20:33:43.703329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.703309    5424 scope.go:122] "RemoveContainer" containerID="1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"
Nov 03 20:33:43.703758 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:43.703734    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d\": container with ID starting with 1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d not found: ID does not exist" containerID="1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"
Nov 03 20:33:43.703859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.703766    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d"} err="failed to get container status \"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d\": rpc error: code = NotFound desc = could not find container \"1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d\": container with ID starting with 1c4cb9cce3de2c21e4f55c82ac9e34309273b1df59fe778d13829ca54f852e0d not found: ID does not exist"
Nov 03 20:33:43.703859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.703784    5424 scope.go:122] "RemoveContainer" containerID="16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"
Nov 03 20:33:43.714045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.714018    5424 scope.go:122] "RemoveContainer" containerID="16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"
Nov 03 20:33:43.714353 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:33:43.714335    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387\": container with ID starting with 16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387 not found: ID does not exist" containerID="16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"
Nov 03 20:33:43.714395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.714362    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387"} err="failed to get container status \"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387\": rpc error: code = NotFound desc = could not find container \"16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387\": container with ID starting with 16ee2fff00a1a3eae61cb91423bef92c90d40ffa15df2858df0c1edde197b387 not found: ID does not exist"
Nov 03 20:33:43.838170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.838059    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:33:43.843686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.843658    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5"]
Nov 03 20:33:43.877888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.877856    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:33:43.880597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.880576    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/simulator-w5-fd01ad-kserve-7cfc74bd-zkml4"]
Nov 03 20:33:43.920340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.920304    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" path="/var/lib/kubelet/pods/a18071f0-b070-4d57-aa55-a6ae470f34b5/volumes"
Nov 03 20:33:43.920806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.920792    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" path="/var/lib/kubelet/pods/f42f85c3-00f2-4335-9556-dc8882d2f4fd/volumes"
Nov 03 20:33:43.921092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.921080    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:33:43.921134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:43.921108    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4"]
Nov 03 20:33:44.225867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.225773    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/distinct-2-w5-fd01ad-kserve-7d9b97f875-lrrw5" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:44.241780 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.241736    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fd01ad/premium-w5-fd01ad-kserve-5c9cdb64b9-pj5n4" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:44.851227 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.851198    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:44.897092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.897063    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\" (UniqueName: \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\") pod \"267a1961-2acb-435e-b8cb-e2937570c03c\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") "
Nov 03 20:33:44.897268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.897107    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\") pod \"267a1961-2acb-435e-b8cb-e2937570c03c\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") "
Nov 03 20:33:44.897268 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.897247    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\") pod \"267a1961-2acb-435e-b8cb-e2937570c03c\" (UID: \"267a1961-2acb-435e-b8cb-e2937570c03c\") "
Nov 03 20:33:44.897512 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.897488    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca" pod "267a1961-2acb-435e-b8cb-e2937570c03c" (UID: "267a1961-2acb-435e-b8cb-e2937570c03c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:44.897734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.897715    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:44.898134 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.898109    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:44.899779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.899755    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds" pod "267a1961-2acb-435e-b8cb-e2937570c03c" (UID: "267a1961-2acb-435e-b8cb-e2937570c03c"). InnerVolumeSpecName "kube-api-access-d62ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:44.899863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.899796    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca" pod "267a1961-2acb-435e-b8cb-e2937570c03c" (UID: "267a1961-2acb-435e-b8cb-e2937570c03c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:44.999466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.998735    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\" (UniqueName: \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\") pod \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") "
Nov 03 20:33:44.999466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.998809    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\") pod \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") "
Nov 03 20:33:44.999466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.998857    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\") pod \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\" (UID: \"772286d7-8bcd-4fb6-a143-ade7c5d4488c\") "
Nov 03 20:33:44.999466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.999271    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267a1961-2acb-435e-b8cb-e2937570c03c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:44.999466 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:44.999296    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62ds\" (UniqueName: \"kubernetes.io/projected/267a1961-2acb-435e-b8cb-e2937570c03c-kube-api-access-d62ds\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:45.001196 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.001150    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca" pod "772286d7-8bcd-4fb6-a143-ade7c5d4488c" (UID: "772286d7-8bcd-4fb6-a143-ade7c5d4488c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:45.002321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.002284    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca" pod "772286d7-8bcd-4fb6-a143-ade7c5d4488c" (UID: "772286d7-8bcd-4fb6-a143-ade7c5d4488c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:45.002586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.002513    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp" pod "772286d7-8bcd-4fb6-a143-ade7c5d4488c" (UID: "772286d7-8bcd-4fb6-a143-ade7c5d4488c"). InnerVolumeSpecName "kube-api-access-6s9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:45.101013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.100975    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9hp\" (UniqueName: \"kubernetes.io/projected/772286d7-8bcd-4fb6-a143-ade7c5d4488c-kube-api-access-6s9hp\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:45.101013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.101015    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:45.101234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.101030    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772286d7-8bcd-4fb6-a143-ade7c5d4488c-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:45.708350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.708312    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76" event={"ID":"772286d7-8bcd-4fb6-a143-ade7c5d4488c","Type":"ContainerDied","Data":"a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064"}
Nov 03 20:33:45.708548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.708366    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91e0ba2f26f7c05170192737cad2218dcdbe1634929e2571ec7c24fcfd78064"
Nov 03 20:33:45.708548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.708320    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"
Nov 03 20:33:45.710047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.710014    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"
Nov 03 20:33:45.710047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.710036    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7" event={"ID":"267a1961-2acb-435e-b8cb-e2937570c03c","Type":"ContainerDied","Data":"a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431"}
Nov 03 20:33:45.710274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.710084    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89c45ada7bb4cf54215323272032aab35bf29b6f106ee581f8a158e89d25431"
Nov 03 20:33:45.920245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:45.920212    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" path="/var/lib/kubelet/pods/245b12cc-7342-4444-8e86-d85ed0e34f5a/volumes"
Nov 03 20:33:46.614198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.614156    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"]
Nov 03 20:33:46.614941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.614919    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main"
Nov 03 20:33:46.615004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.614945    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main"
Nov 03 20:33:46.615004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.614991    5424 cpu_manager.go:414] "containerMap: removing container" podUID="772286d7-8bcd-4fb6-a143-ade7c5d4488c" containerName="revoke-keys"
Nov 03 20:33:46.615004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615000    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772286d7-8bcd-4fb6-a143-ade7c5d4488c" containerName="revoke-keys"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615018    5424 cpu_manager.go:414] "containerMap: removing container" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615025    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615037    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615045    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615078    5424 cpu_manager.go:414] "containerMap: removing container" podUID="267a1961-2acb-435e-b8cb-e2937570c03c" containerName="revoke-keys"
Nov 03 20:33:46.615102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615086    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267a1961-2acb-435e-b8cb-e2937570c03c" containerName="revoke-keys"
Nov 03 20:33:46.615282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615198    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18071f0-b070-4d57-aa55-a6ae470f34b5" containerName="main"
Nov 03 20:33:46.615282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615208    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="267a1961-2acb-435e-b8cb-e2937570c03c" containerName="revoke-keys"
Nov 03 20:33:46.615282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615226    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42f85c3-00f2-4335-9556-dc8882d2f4fd" containerName="main"
Nov 03 20:33:46.615282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615242    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="772286d7-8bcd-4fb6-a143-ade7c5d4488c" containerName="revoke-keys"
Nov 03 20:33:46.615282 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.615252    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="245b12cc-7342-4444-8e86-d85ed0e34f5a" containerName="main"
Nov 03 20:33:46.620157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.620133    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.654561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.647854    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:33:46.654561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.648623    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:33:46.680560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.668633    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"]
Nov 03 20:33:46.680560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.673393    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"]
Nov 03 20:33:46.694555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.689342    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.698272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.695716    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"]
Nov 03 20:33:46.722985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.722938    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnds\" (UniqueName: \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.723169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.723011    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.723169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.723095    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.723169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.723148    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.723334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.723207    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcrr\" (UniqueName: \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.723334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.723288    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.824509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824444    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.824750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824577    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.824750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824652    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcrr\" (UniqueName: \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.824750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824736    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.824943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824830    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnds\" (UniqueName: \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.824943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.824881    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.825458 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.825433    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.826221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.826196    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.827405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.827375    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.827505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.827414    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.845776 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.845740    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnds\" (UniqueName: \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\") pod \"maas-api-revoke-sub-008f4600-5mbx7\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:46.845942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.845920    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcrr\" (UniqueName: \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\") pod \"maas-api-revoke-sub-2e194f11-lj5n6\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:46.996382 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:46.996349    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:47.013825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.013794    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:47.328697 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.327137    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"]
Nov 03 20:33:47.352052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.352008    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"]
Nov 03 20:33:47.355690 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:47.355661    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c27e32_3acd_410b_bcfe_f82eb4dd24e0.slice/crio-8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d WatchSource:0}: Error finding container 8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d: Status 404 returned error can't find the container with id 8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d
Nov 03 20:33:47.745283 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.745241    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7" event={"ID":"03f4f692-d011-4003-b79e-e16f1b9d2d4d","Type":"ContainerStarted","Data":"6b56d5cce4968e3a1d3f3fc0027b245d7e4f935aaba15bb877677f74a6625022"}
Nov 03 20:33:47.747877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.747838    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6" event={"ID":"44c27e32-3acd-410b-bcfe-f82eb4dd24e0","Type":"ContainerStarted","Data":"fc4250b276e82648fe82bc00ceec0f16b7ba5238de5466d1965070905eb80aa7"}
Nov 03 20:33:47.748021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.747898    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6" event={"ID":"44c27e32-3acd-410b-bcfe-f82eb4dd24e0","Type":"ContainerStarted","Data":"8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d"}
Nov 03 20:33:47.795176 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:47.795116    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6" podStartSLOduration=1.795099467 podStartE2EDuration="1.795099467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:47.79015269 +0000 UTC m=+1769.999168688" watchObservedRunningTime="2026-11-03 20:33:47.795099467 +0000 UTC m=+1770.004115510"
Nov 03 20:33:48.760061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:48.760015    5424 generic.go:356] "Generic (PLEG): container finished" podID="03f4f692-d011-4003-b79e-e16f1b9d2d4d" containerID="53de101d0614481b5594fca464559ea18d89567bdc061510e63b52fd37b8f384" exitCode=0
Nov 03 20:33:48.760571 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:48.760083    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7" event={"ID":"03f4f692-d011-4003-b79e-e16f1b9d2d4d","Type":"ContainerDied","Data":"53de101d0614481b5594fca464559ea18d89567bdc061510e63b52fd37b8f384"}
Nov 03 20:33:48.763022 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:48.762994    5424 generic.go:356] "Generic (PLEG): container finished" podID="44c27e32-3acd-410b-bcfe-f82eb4dd24e0" containerID="fc4250b276e82648fe82bc00ceec0f16b7ba5238de5466d1965070905eb80aa7" exitCode=0
Nov 03 20:33:48.763178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:48.763066    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6" event={"ID":"44c27e32-3acd-410b-bcfe-f82eb4dd24e0","Type":"ContainerDied","Data":"fc4250b276e82648fe82bc00ceec0f16b7ba5238de5466d1965070905eb80aa7"}
Nov 03 20:33:49.422048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.422009    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"]
Nov 03 20:33:49.426123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.426100    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.444327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.444297    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"]
Nov 03 20:33:49.497179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.497144    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.497378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.497282    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.497378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.497323    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2rb\" (UniqueName: \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.598632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.598594    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.598632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.598647    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2rb\" (UniqueName: \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.598914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.598760    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.599464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.599443    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.601341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.601320    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.618355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.618320    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2rb\" (UniqueName: \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\") pod \"maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:49.740910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:49.740871    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:50.000336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.000256    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:50.002520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.002498    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:50.059244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.059206    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"]
Nov 03 20:33:50.059826 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:50.059790    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7905e3d5_a316_45c2_bcf0_42b43d7fa2a0.slice/crio-f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c WatchSource:0}: Error finding container f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c: Status 404 returned error can't find the container with id f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c
Nov 03 20:33:50.104957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.104918    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\") pod \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") "
Nov 03 20:33:50.105120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105005    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\") pod \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") "
Nov 03 20:33:50.105120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105044    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\" (UniqueName: \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\") pod \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") "
Nov 03 20:33:50.105259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105131    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\") pod \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\" (UID: \"03f4f692-d011-4003-b79e-e16f1b9d2d4d\") "
Nov 03 20:33:50.105259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105159    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\" (UniqueName: \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\") pod \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") "
Nov 03 20:33:50.105259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105218    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\") pod \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\" (UID: \"44c27e32-3acd-410b-bcfe-f82eb4dd24e0\") "
Nov 03 20:33:50.105433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105403    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca" pod "03f4f692-d011-4003-b79e-e16f1b9d2d4d" (UID: "03f4f692-d011-4003-b79e-e16f1b9d2d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:50.105867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105839    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.106114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.105977    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca" pod "44c27e32-3acd-410b-bcfe-f82eb4dd24e0" (UID: "44c27e32-3acd-410b-bcfe-f82eb4dd24e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:50.107715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.107683    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca" pod "03f4f692-d011-4003-b79e-e16f1b9d2d4d" (UID: "03f4f692-d011-4003-b79e-e16f1b9d2d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:50.107820 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.107794    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr" pod "44c27e32-3acd-410b-bcfe-f82eb4dd24e0" (UID: "44c27e32-3acd-410b-bcfe-f82eb4dd24e0"). InnerVolumeSpecName "kube-api-access-4wcrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:50.107933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.107915    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds" pod "03f4f692-d011-4003-b79e-e16f1b9d2d4d" (UID: "03f4f692-d011-4003-b79e-e16f1b9d2d4d"). InnerVolumeSpecName "kube-api-access-7rnds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:50.109066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.109049    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca" pod "44c27e32-3acd-410b-bcfe-f82eb4dd24e0" (UID: "44c27e32-3acd-410b-bcfe-f82eb4dd24e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:50.207180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.207136    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f4f692-d011-4003-b79e-e16f1b9d2d4d-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.207180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.207172    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wcrr\" (UniqueName: \"kubernetes.io/projected/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-kube-api-access-4wcrr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.207180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.207184    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.207180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.207192    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c27e32-3acd-410b-bcfe-f82eb4dd24e0-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.207467 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.207201    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnds\" (UniqueName: \"kubernetes.io/projected/03f4f692-d011-4003-b79e-e16f1b9d2d4d-kube-api-access-7rnds\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:50.781462 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.781427    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"
Nov 03 20:33:50.781651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.781483    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6" event={"ID":"44c27e32-3acd-410b-bcfe-f82eb4dd24e0","Type":"ContainerDied","Data":"8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d"}
Nov 03 20:33:50.781651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.781573    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d251672aaee1df26582fe9b1ecd1a2aab3bb2814e75fe3c9b7e2af84cec372d"
Nov 03 20:33:50.784033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.783740    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg" event={"ID":"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0","Type":"ContainerStarted","Data":"32589e7d03729e7848b760c28b2e4f370970e4c75e315d50c4d5c73c8f668698"}
Nov 03 20:33:50.784033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.783786    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg" event={"ID":"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0","Type":"ContainerStarted","Data":"f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c"}
Nov 03 20:33:50.785344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.785318    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7" event={"ID":"03f4f692-d011-4003-b79e-e16f1b9d2d4d","Type":"ContainerDied","Data":"6b56d5cce4968e3a1d3f3fc0027b245d7e4f935aaba15bb877677f74a6625022"}
Nov 03 20:33:50.785344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.785337    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"
Nov 03 20:33:50.785482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.785364    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b56d5cce4968e3a1d3f3fc0027b245d7e4f935aaba15bb877677f74a6625022"
Nov 03 20:33:50.823284 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:50.823233    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg" podStartSLOduration=1.8232197060000002 podStartE2EDuration="1.823219706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:50.821776916 +0000 UTC m=+1773.030792905" watchObservedRunningTime="2026-11-03 20:33:50.823219706 +0000 UTC m=+1773.032235702"
Nov 03 20:33:51.793895 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:51.793852    5424 generic.go:356] "Generic (PLEG): container finished" podID="7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" containerID="32589e7d03729e7848b760c28b2e4f370970e4c75e315d50c4d5c73c8f668698" exitCode=0
Nov 03 20:33:51.794438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:51.793943    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg" event={"ID":"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0","Type":"ContainerDied","Data":"32589e7d03729e7848b760c28b2e4f370970e4c75e315d50c4d5c73c8f668698"}
Nov 03 20:33:52.944385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:52.944353    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:53.045971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.045932    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\") pod \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") "
Nov 03 20:33:53.046187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.046042    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\" (UniqueName: \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\") pod \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") "
Nov 03 20:33:53.046187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.046152    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\") pod \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\" (UID: \"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0\") "
Nov 03 20:33:53.046415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.046389    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca" pod "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" (UID: "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:53.048570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.048505    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca" pod "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" (UID: "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:53.048570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.048558    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb" pod "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" (UID: "7905e3d5-a316-45c2-bcf0-42b43d7fa2a0"). InnerVolumeSpecName "kube-api-access-dz2rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:53.147223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.147182    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2rb\" (UniqueName: \"kubernetes.io/projected/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-kube-api-access-dz2rb\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:53.147223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.147218    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:53.147223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.147229    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:33:53.809200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.809170    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"
Nov 03 20:33:53.809200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.809195    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg" event={"ID":"7905e3d5-a316-45c2-bcf0-42b43d7fa2a0","Type":"ContainerDied","Data":"f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c"}
Nov 03 20:33:53.809444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:53.809243    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54aaca8fde8643ff6c258629f9c7553d519ba0d760b0c8aff015e7fee18c10c"
Nov 03 20:33:54.165441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.165343    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.627680    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628469    5424 cpu_manager.go:414] "containerMap: removing container" podUID="03f4f692-d011-4003-b79e-e16f1b9d2d4d" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628492    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f4f692-d011-4003-b79e-e16f1b9d2d4d" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628593    5424 cpu_manager.go:414] "containerMap: removing container" podUID="44c27e32-3acd-410b-bcfe-f82eb4dd24e0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628602    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c27e32-3acd-410b-bcfe-f82eb4dd24e0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628617    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628625    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628742    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c27e32-3acd-410b-bcfe-f82eb4dd24e0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628756    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" containerName="revoke-keys"
Nov 03 20:33:54.636383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.628765    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f4f692-d011-4003-b79e-e16f1b9d2d4d" containerName="revoke-keys"
Nov 03 20:33:54.640633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.639498    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:33:54.640633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.639652    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.667244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.667185    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.667584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.667558    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.668722 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.668698    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.668899 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.668881    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.669729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.669689    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.669847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.669736    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.669847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.669803    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwt9\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.669981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.669849    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.669981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.669945    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.697861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.697827    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4b95f3-openshift-default-dockercfg-9rljv\""
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.775802    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.775871    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.775938    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776022    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776124    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwt9\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776191    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776274    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776293    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776374    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.776439    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.777556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.777499    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.778334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.777999    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.778334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.778101    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.778334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.778264    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.786017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.785949    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.788576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.788499    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.798995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.798961    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.799605 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.799573    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwt9\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\") pod \"e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") " pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:54.972615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:54.972480    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:55.317815 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:33:55.317773    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb749d69_3bdb_4edf_8a8e_971697c5d109.slice/crio-9f92e505ac7687bf25fd1d9a0cafbad9e99d3ac30a570d255ef82d2732c4003f WatchSource:0}: Error finding container 9f92e505ac7687bf25fd1d9a0cafbad9e99d3ac30a570d255ef82d2732c4003f: Status 404 returned error can't find the container with id 9f92e505ac7687bf25fd1d9a0cafbad9e99d3ac30a570d255ef82d2732c4003f
Nov 03 20:33:55.321311 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.321267    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:33:55.328304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.328260    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:55.328411 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.328380    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:55.328456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.328424    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:55.832611 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.832575    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerStarted","Data":"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"}
Nov 03 20:33:55.832795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.832633    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerStarted","Data":"9f92e505ac7687bf25fd1d9a0cafbad9e99d3ac30a570d255ef82d2732c4003f"}
Nov 03 20:33:55.884517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.882806    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podStartSLOduration=1.882782047 podStartE2EDuration="1.882782047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:55.879722442 +0000 UTC m=+1778.088738440" watchObservedRunningTime="2026-11-03 20:33:55.882782047 +0000 UTC m=+1778.091798045"
Nov 03 20:33:55.973712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:55.973670    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:56.536355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:56.536321    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"]
Nov 03 20:33:56.548907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:56.548870    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-9e1bcb-z4cls"]
Nov 03 20:33:56.979795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:56.979717    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:57.848321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:57.848286    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:57.849490 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:57.849465    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:33:57.922665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:57.922627    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b" path="/var/lib/kubelet/pods/5c5a8e6e-2d2b-4587-bd56-1ba27305ed6b/volumes"
Nov 03 20:33:59.957799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:33:59.957755    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:34:04.029522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:04.029483    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"]
Nov 03 20:34:04.041859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:04.041817    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e254cb6-8nd85"]
Nov 03 20:34:04.044956 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:04.044932    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"]
Nov 03 20:34:04.047499 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:04.047475    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51b1e1f-hg8pq"]
Nov 03 20:34:04.958724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:04.958660    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing" containerID="cri-o://e4f9043ef57dffb6f1e6677b8a2440fa55425f4fd2f0d45f814c20a80d63e893" gracePeriod=25
Nov 03 20:34:05.777832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:05.777792    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:34:05.778376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:05.778053    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy" containerID="cri-o://fc99d91b67dfada843d8b70e38a1d9339c939c600d51997e9807f31dd2a210e5" gracePeriod=30
Nov 03 20:34:05.928181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:05.928142    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8638cb-8f6a-4b56-8e3c-0df46816b0d6" path="/var/lib/kubelet/pods/7e8638cb-8f6a-4b56-8e3c-0df46816b0d6/volumes"
Nov 03 20:34:05.928627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:05.928613    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5833dc4-263d-4829-b939-6d83454f6b96" path="/var/lib/kubelet/pods/d5833dc4-263d-4829-b939-6d83454f6b96/volumes"
Nov 03 20:34:06.169813 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.169729    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"]
Nov 03 20:34:06.174914 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.174890    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.191581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.191549    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:34:06.191761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.191676    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:34:06.194646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.194624    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"]
Nov 03 20:34:06.305748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.305699    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvd7\" (UniqueName: \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.306003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.305976    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.306100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.306080    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.407313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.407267    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.407554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.407361    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.407554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.407454    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvd7\" (UniqueName: \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.408109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.408084    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.410187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.410151    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.429769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.429692    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvd7\" (UniqueName: \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\") pod \"maas-api-revoke-sub-d7b51689-p8bt8\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.491337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.491303    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:06.820244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.820214    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"]
Nov 03 20:34:06.942501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:06.942467    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8" event={"ID":"b6216744-8e3b-4f88-b670-e5ee474624e0","Type":"ContainerStarted","Data":"f415a9bf8a0de8b6ac3093be223374c133bc9929449f4ce576c22645a9b823c1"}
Nov 03 20:34:07.020910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.020878    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"]
Nov 03 20:34:07.025787 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.025753    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9973d9-2sj6l"]
Nov 03 20:34:07.655179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.655071    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"]
Nov 03 20:34:07.667298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.667268    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fd01ad-4wrhs"]
Nov 03 20:34:07.920637 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.920513    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8f7cbf-ba76-4d64-b24d-32a8a61d849a" path="/var/lib/kubelet/pods/df8f7cbf-ba76-4d64-b24d-32a8a61d849a/volumes"
Nov 03 20:34:07.921146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.921127    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e3ee72-11e5-470c-b9f0-8ecb392b5663" path="/var/lib/kubelet/pods/f3e3ee72-11e5-470c-b9f0-8ecb392b5663/volumes"
Nov 03 20:34:07.953026 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.952994    5424 generic.go:356] "Generic (PLEG): container finished" podID="b6216744-8e3b-4f88-b670-e5ee474624e0" containerID="967829728ad8fcd1c537216785a76375ed557cfebfa877ba6f23d3bd6671987a" exitCode=0
Nov 03 20:34:07.953232 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:07.953078    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8" event={"ID":"b6216744-8e3b-4f88-b670-e5ee474624e0","Type":"ContainerDied","Data":"967829728ad8fcd1c537216785a76375ed557cfebfa877ba6f23d3bd6671987a"}
Nov 03 20:34:09.121449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.121418    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:09.181491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.181455    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\" (UniqueName: \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\") pod \"b6216744-8e3b-4f88-b670-e5ee474624e0\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") "
Nov 03 20:34:09.181740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.181501    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\") pod \"b6216744-8e3b-4f88-b670-e5ee474624e0\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") "
Nov 03 20:34:09.181740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.181646    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\") pod \"b6216744-8e3b-4f88-b670-e5ee474624e0\" (UID: \"b6216744-8e3b-4f88-b670-e5ee474624e0\") "
Nov 03 20:34:09.182009 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.181974    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca" pod "b6216744-8e3b-4f88-b670-e5ee474624e0" (UID: "b6216744-8e3b-4f88-b670-e5ee474624e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:09.184057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.184030    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca" pod "b6216744-8e3b-4f88-b670-e5ee474624e0" (UID: "b6216744-8e3b-4f88-b670-e5ee474624e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:09.184374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.184348    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7" pod "b6216744-8e3b-4f88-b670-e5ee474624e0" (UID: "b6216744-8e3b-4f88-b670-e5ee474624e0"). InnerVolumeSpecName "kube-api-access-jpvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:09.189522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.189490    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main" containerID="cri-o://b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280" gracePeriod=45
Nov 03 20:34:09.282714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.282678    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvd7\" (UniqueName: \"kubernetes.io/projected/b6216744-8e3b-4f88-b670-e5ee474624e0-kube-api-access-jpvd7\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.282714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.282711    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.282714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.282721    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6216744-8e3b-4f88-b670-e5ee474624e0-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.431355 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.431326    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:34:09.485349 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485315    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.485696 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485661    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.485855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485732    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.485855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485742    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.485977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485907    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.485977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485947    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.485977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.485967    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.486317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.486013    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\" (UniqueName: \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\") pod \"1c126010-cc33-4372-b223-71335546706b\" (UID: \"1c126010-cc33-4372-b223-71335546706b\") "
Nov 03 20:34:09.486700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.486672    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.487086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.487063    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.487185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.487088    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.487185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.487100    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.488465 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.488444    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.488781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.488758    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:09.488975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.488942    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw" pod "1c126010-cc33-4372-b223-71335546706b" (UID: "1c126010-cc33-4372-b223-71335546706b"). InnerVolumeSpecName "kube-api-access-64xfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:09.589025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.588985    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xfw\" (UniqueName: \"kubernetes.io/projected/1c126010-cc33-4372-b223-71335546706b-kube-api-access-64xfw\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.589025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.589023    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c126010-cc33-4372-b223-71335546706b-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.589025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.589033    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c126010-cc33-4372-b223-71335546706b-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.969246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.969210    5424 generic.go:356] "Generic (PLEG): container finished" podID="1c126010-cc33-4372-b223-71335546706b" containerID="b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280" exitCode=0
Nov 03 20:34:09.969438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.969268    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"
Nov 03 20:34:09.969438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.969268    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" event={"ID":"1c126010-cc33-4372-b223-71335546706b","Type":"ContainerDied","Data":"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"}
Nov 03 20:34:09.969438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.969331    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp" event={"ID":"1c126010-cc33-4372-b223-71335546706b","Type":"ContainerDied","Data":"80f036569b14b2ce6f857399a180fe397c1504ecd919a78c3c7b3b3d58397984"}
Nov 03 20:34:09.969438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.969354    5424 scope.go:122] "RemoveContainer" containerID="b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"
Nov 03 20:34:09.972164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.971954    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8" event={"ID":"b6216744-8e3b-4f88-b670-e5ee474624e0","Type":"ContainerDied","Data":"f415a9bf8a0de8b6ac3093be223374c133bc9929449f4ce576c22645a9b823c1"}
Nov 03 20:34:09.972164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.972005    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f415a9bf8a0de8b6ac3093be223374c133bc9929449f4ce576c22645a9b823c1"
Nov 03 20:34:09.972164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.972064    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"
Nov 03 20:34:09.982124 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.982105    5424 scope.go:122] "RemoveContainer" containerID="b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"
Nov 03 20:34:09.982504 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:09.982477    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280\": container with ID starting with b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280 not found: ID does not exist" containerID="b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"
Nov 03 20:34:09.982630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:09.982512    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280"} err="failed to get container status \"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280\": rpc error: code = NotFound desc = could not find container \"b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280\": container with ID starting with b6a29a3a994192611a129ef6c03b497cd74960402140abb18fa264af408ca280 not found: ID does not exist"
Nov 03 20:34:10.022004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:10.021963    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:34:10.034577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:10.034504    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/premium-w0-e40816-kserve-786ff44b4-7h4pp"]
Nov 03 20:34:10.983775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:10.983738    5424 generic.go:356] "Generic (PLEG): container finished" podID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerID="fc99d91b67dfada843d8b70e38a1d9339c939c600d51997e9807f31dd2a210e5" exitCode=0
Nov 03 20:34:10.984279 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:10.983816    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" event={"ID":"e29b35fb-7a74-4fe2-9da4-96b0e5a94966","Type":"ContainerDied","Data":"fc99d91b67dfada843d8b70e38a1d9339c939c600d51997e9807f31dd2a210e5"}
Nov 03 20:34:11.069922 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.069896    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:34:11.106923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.106889    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107045    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107082    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107112    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107173    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107202    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107200    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.107416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107308    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107415    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107473    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\") pod \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\" (UID: \"e29b35fb-7a74-4fe2-9da4-96b0e5a94966\") "
Nov 03 20:34:11.107685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107556    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.107760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.107740    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.108199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.108097    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.108199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.108123    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.108199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.108137    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.108653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.108245    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.108653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.108520    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:11.109854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.109827    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "kube-api-access-27945". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:11.110084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.110063    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.110241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.110217    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:11.110241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.110219    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo" pod "e29b35fb-7a74-4fe2-9da4-96b0e5a94966" (UID: "e29b35fb-7a74-4fe2-9da4-96b0e5a94966"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:11.209606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209564    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.209606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209609    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.209825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209625    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27945\" (UniqueName: \"kubernetes.io/projected/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-kube-api-access-27945\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.209825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209640    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.209825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209654    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.209825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.209667    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29b35fb-7a74-4fe2-9da4-96b0e5a94966-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.866094 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.866055    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:34:11.920044 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.920012    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c126010-cc33-4372-b223-71335546706b" path="/var/lib/kubelet/pods/1c126010-cc33-4372-b223-71335546706b/volumes"
Nov 03 20:34:11.992125 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.992099    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"
Nov 03 20:34:11.992614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.992138    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9" event={"ID":"e29b35fb-7a74-4fe2-9da4-96b0e5a94966","Type":"ContainerDied","Data":"22a4af44746697a191308852c85d4f6e4d86ce66b02b252255bba688442def07"}
Nov 03 20:34:11.992614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:11.992207    5424 scope.go:122] "RemoveContainer" containerID="fc99d91b67dfada843d8b70e38a1d9339c939c600d51997e9807f31dd2a210e5"
Nov 03 20:34:12.058617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:12.058583    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:34:12.070988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:12.070956    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-52gz9"]
Nov 03 20:34:12.286954 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:12.286915    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:34:13.920985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:13.920946    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" path="/var/lib/kubelet/pods/e29b35fb-7a74-4fe2-9da4-96b0e5a94966/volumes"
Nov 03 20:34:14.288943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:14.288906    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:34:14.289226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:14.289200    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy" containerID="cri-o://0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7" gracePeriod=30
Nov 03 20:34:15.021056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:15.021017    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:34:15.021587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:15.021272    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy" containerID="cri-o://928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c" gracePeriod=30
Nov 03 20:34:15.033422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:15.033385    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:34:15.033646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:15.033622    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" containerID="cri-o://02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f" gracePeriod=30
Nov 03 20:34:19.156057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.156017    5424 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:34:19.156478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.156078    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:34:19.558861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.558833    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:34:19.616298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616261    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616326    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616392    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616423    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616463    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616492    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616709    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616785    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616825    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\") pod \"c92715e3-5479-4173-86af-e8d73bbb1570\" (UID: \"c92715e3-5479-4173-86af-e8d73bbb1570\") "
Nov 03 20:34:19.616874 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.616841    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.617519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.617326    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.617891 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.617702    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.618171 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.618017    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.618442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.618279    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.618728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.618702    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:19.620014 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.619975    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:19.620128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.620006    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.620128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.620101    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "kube-api-access-7rv8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:19.621168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.621129    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo" pod "c92715e3-5479-4173-86af-e8d73bbb1570" (UID: "c92715e3-5479-4173-86af-e8d73bbb1570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727444    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92715e3-5479-4173-86af-e8d73bbb1570-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727494    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92715e3-5479-4173-86af-e8d73bbb1570-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727508    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727523    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727558    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.727565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727571    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rv8p\" (UniqueName: \"kubernetes.io/projected/c92715e3-5479-4173-86af-e8d73bbb1570-kube-api-access-7rv8p\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.728003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727584    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.728003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:19.727598    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92715e3-5479-4173-86af-e8d73bbb1570-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.054394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.054360    5424 generic.go:356] "Generic (PLEG): container finished" podID="c92715e3-5479-4173-86af-e8d73bbb1570" containerID="0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7" exitCode=0
Nov 03 20:34:20.054576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.054407    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"
Nov 03 20:34:20.054576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.054436    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" event={"ID":"c92715e3-5479-4173-86af-e8d73bbb1570","Type":"ContainerDied","Data":"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"}
Nov 03 20:34:20.054576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.054490    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn" event={"ID":"c92715e3-5479-4173-86af-e8d73bbb1570","Type":"ContainerDied","Data":"6694843f00f8b5cd477785124609696d972f5b26d6bab8b8e4a3bbfe4c336f5d"}
Nov 03 20:34:20.054576 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.054518    5424 scope.go:122] "RemoveContainer" containerID="0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"
Nov 03 20:34:20.165085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.165040    5424 scope.go:122] "RemoveContainer" containerID="0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"
Nov 03 20:34:20.165601 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:20.165501    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7\": container with ID starting with 0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7 not found: ID does not exist" containerID="0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"
Nov 03 20:34:20.165601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.165552    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7"} err="failed to get container status \"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7\": rpc error: code = NotFound desc = could not find container \"0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7\": container with ID starting with 0da0bc8476fcde929b863dc36b4f2393bf0f2a045d76154dc56bd2b614db92a7 not found: ID does not exist"
Nov 03 20:34:20.197507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.197071    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:34:20.208210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.207872    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-9q2wn"]
Nov 03 20:34:20.299750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.299722    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:34:20.361967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.361938    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:34:20.434854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.434819    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.434869    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.434915    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.434944    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.434973    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435010    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435086    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435122    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435163    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435210    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435236    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435270    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435304    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435343    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435378    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435460    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435450    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435506    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435506    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435591    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\") pod \"e8bc23ca-27dd-44cc-936a-5372b991894b\" (UID: \"e8bc23ca-27dd-44cc-936a-5372b991894b\") "
Nov 03 20:34:20.435742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435674    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\") pod \"3c910095-27d0-4a7f-87de-908bdb8b94c7\" (UID: \"3c910095-27d0-4a7f-87de-908bdb8b94c7\") "
Nov 03 20:34:20.436190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.435785    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:20.436190 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436033    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.436318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436191    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.436318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436238    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.436318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436252    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c910095-27d0-4a7f-87de-908bdb8b94c7-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.436318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436266    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.436318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436280    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.436642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436355    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:20.436916 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436888    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.437261 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.436993    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.438152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.438118    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "kube-api-access-x4tnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.438385 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.438361    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.438617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.438593    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.439181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.439152    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.441337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.441300    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:20.441506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.441481    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.441734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.441585    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.441888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.441863    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo" pod "3c910095-27d0-4a7f-87de-908bdb8b94c7" (UID: "3c910095-27d0-4a7f-87de-908bdb8b94c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:20.442926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.442899    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "kube-api-access-cdvj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.443231 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.443208    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token" pod "e8bc23ca-27dd-44cc-936a-5372b991894b" (UID: "e8bc23ca-27dd-44cc-936a-5372b991894b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.537006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.536971    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537006 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537002    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tnk\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-kube-api-access-x4tnk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537017    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvj4\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-kube-api-access-cdvj4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537030    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bc23ca-27dd-44cc-936a-5372b991894b-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537042    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537052    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537062    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537073    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537083    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537094    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537107    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537119    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bc23ca-27dd-44cc-936a-5372b991894b-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.537218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.537129    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c910095-27d0-4a7f-87de-908bdb8b94c7-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.899131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899084    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:34:20.899676 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899657    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899678    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899707    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b6216744-8e3b-4f88-b670-e5ee474624e0" containerName="revoke-keys"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899714    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6216744-8e3b-4f88-b670-e5ee474624e0" containerName="revoke-keys"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899732    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899739    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899752    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899763    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899781    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899788    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899801    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy"
Nov 03 20:34:20.899812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899807    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899921    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6216744-8e3b-4f88-b670-e5ee474624e0" containerName="revoke-keys"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899936    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" containerName="istio-proxy"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899947    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerName="istio-proxy"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899957    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c126010-cc33-4372-b223-71335546706b" containerName="main"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899966    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b35fb-7a74-4fe2-9da4-96b0e5a94966" containerName="istio-proxy"
Nov 03 20:34:20.900344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.899978    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerName="istio-proxy"
Nov 03 20:34:20.909568 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.905967    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:20.924959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.924928    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:34:20.949049 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:20.949015    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1d159a-openshift-default-dockercfg-db82z\""
Nov 03 20:34:21.042583 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.042513    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.042750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.042695    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.042750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.042717    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.042750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.042740    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8sv2\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.042950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.042923    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.043013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.043001    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.043059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.043030    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.043059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.043056    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.043165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.043073    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.063825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.063789    5424 generic.go:356] "Generic (PLEG): container finished" podID="e8bc23ca-27dd-44cc-936a-5372b991894b" containerID="928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c" exitCode=0
Nov 03 20:34:21.063985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.063854    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"
Nov 03 20:34:21.063985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.063875    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" event={"ID":"e8bc23ca-27dd-44cc-936a-5372b991894b","Type":"ContainerDied","Data":"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"}
Nov 03 20:34:21.063985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.063949    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72" event={"ID":"e8bc23ca-27dd-44cc-936a-5372b991894b","Type":"ContainerDied","Data":"a90f6e66ea974002b452821b38f87e61e0c054dd2e5c055c3fdb8fc6e3573c98"}
Nov 03 20:34:21.063985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.063980    5424 scope.go:122] "RemoveContainer" containerID="928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"
Nov 03 20:34:21.067451 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.067421    5424 generic.go:356] "Generic (PLEG): container finished" podID="3c910095-27d0-4a7f-87de-908bdb8b94c7" containerID="02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f" exitCode=0
Nov 03 20:34:21.067623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.067484    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" event={"ID":"3c910095-27d0-4a7f-87de-908bdb8b94c7","Type":"ContainerDied","Data":"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"}
Nov 03 20:34:21.067623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.067555    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"
Nov 03 20:34:21.067623 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.067564    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr" event={"ID":"3c910095-27d0-4a7f-87de-908bdb8b94c7","Type":"ContainerDied","Data":"bc49a9c123d54b353fcb61ccbc3ff6ed309b5a70c869655c84d75c52156e689a"}
Nov 03 20:34:21.083585 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.083070    5424 scope.go:122] "RemoveContainer" containerID="928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"
Nov 03 20:34:21.084347 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:21.084317    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c\": container with ID starting with 928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c not found: ID does not exist" containerID="928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"
Nov 03 20:34:21.084516 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.084484    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c"} err="failed to get container status \"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c\": rpc error: code = NotFound desc = could not find container \"928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c\": container with ID starting with 928eac131c56f6d8f796a0f1a3cba2361ecbbebc22aa84666cdaf01573cf5c8c not found: ID does not exist"
Nov 03 20:34:21.084660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.084519    5424 scope.go:122] "RemoveContainer" containerID="02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"
Nov 03 20:34:21.097511 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.097490    5424 scope.go:122] "RemoveContainer" containerID="02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"
Nov 03 20:34:21.097899 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:21.097879    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f\": container with ID starting with 02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f not found: ID does not exist" containerID="02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"
Nov 03 20:34:21.097959 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.097911    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f"} err="failed to get container status \"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f\": rpc error: code = NotFound desc = could not find container \"02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f\": container with ID starting with 02b45bfa6ba886ecc826122f8ad1ef290a16c9ab317211ff67c583ef04e6574f not found: ID does not exist"
Nov 03 20:34:21.122362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.122131    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:34:21.125459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.125431    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5kp72"]
Nov 03 20:34:21.144278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144240    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144279    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144316    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8sv2\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144496    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144610    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144644    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144676    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144696    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144810 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144737    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.144940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144916    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.145003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.144982    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.145108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.145059    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.145108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.145070    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.145260 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.145201    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.147034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.147003    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.147114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.147050    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.159746 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.159684    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:34:21.165494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.165465    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-5mjmr"]
Nov 03 20:34:21.166001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.165982    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.166087 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.166068    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8sv2\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.224052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.224014    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:21.332656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.332624    5424 scope.go:122] "RemoveContainer" containerID="404bd68d538fbabea7ffc54bc0a4e0e63db4e0e523d2eb8c83f1cdd2a8fa8d7a"
Nov 03 20:34:21.344321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.344293    5424 scope.go:122] "RemoveContainer" containerID="c6f6063ca87a0d1d877a243201f04434ed2e27099686ef0850b0de12c01c32f8"
Nov 03 20:34:21.541681 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:34:21.541644    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34081f6f_69f1_4a88_a2df_06bc6b0e2c72.slice/crio-06c4a6ac66bbd641de3f589afdca507d37bc0259b8942c13a0136445609749d5 WatchSource:0}: Error finding container 06c4a6ac66bbd641de3f589afdca507d37bc0259b8942c13a0136445609749d5: Status 404 returned error can't find the container with id 06c4a6ac66bbd641de3f589afdca507d37bc0259b8942c13a0136445609749d5
Nov 03 20:34:21.542138 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.542118    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:34:21.544296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.544259    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:21.544404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.544328    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:21.544404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.544357    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:21.865658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.865610    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:34:21.920090 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.920058    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c910095-27d0-4a7f-87de-908bdb8b94c7" path="/var/lib/kubelet/pods/3c910095-27d0-4a7f-87de-908bdb8b94c7/volumes"
Nov 03 20:34:21.920562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.920548    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92715e3-5479-4173-86af-e8d73bbb1570" path="/var/lib/kubelet/pods/c92715e3-5479-4173-86af-e8d73bbb1570/volumes"
Nov 03 20:34:21.920946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:21.920935    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bc23ca-27dd-44cc-936a-5372b991894b" path="/var/lib/kubelet/pods/e8bc23ca-27dd-44cc-936a-5372b991894b/volumes"
Nov 03 20:34:22.076288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.076243    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" event={"ID":"34081f6f-69f1-4a88-a2df-06bc6b0e2c72","Type":"ContainerStarted","Data":"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"}
Nov 03 20:34:22.076486 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.076306    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" event={"ID":"34081f6f-69f1-4a88-a2df-06bc6b0e2c72","Type":"ContainerStarted","Data":"06c4a6ac66bbd641de3f589afdca507d37bc0259b8942c13a0136445609749d5"}
Nov 03 20:34:22.122617 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.121007    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podStartSLOduration=2.120987269 podStartE2EDuration="2.120987269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:22.118259904 +0000 UTC m=+1804.327275907" watchObservedRunningTime="2026-11-03 20:34:22.120987269 +0000 UTC m=+1804.330003270"
Nov 03 20:34:22.225250 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.224920    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:22.226796 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.226762    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:22.227052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:22.227026    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:23.229559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:23.227074    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:23.229559 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:23.227141    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:24.231097 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:24.231048    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:24.231807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:24.231768    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:25.145628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.145415    5424 generic.go:356] "Generic (PLEG): container finished" podID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerID="e4f9043ef57dffb6f1e6677b8a2440fa55425f4fd2f0d45f814c20a80d63e893" exitCode=0
Nov 03 20:34:25.145628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.145500    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" event={"ID":"3968f1ea-e378-4dda-bbb2-d83835be0d92","Type":"ContainerDied","Data":"e4f9043ef57dffb6f1e6677b8a2440fa55425f4fd2f0d45f814c20a80d63e893"}
Nov 03 20:34:25.243575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.233248    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:25.243575 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.233349    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:25.257807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.257775    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:34:25.395624 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.395580    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\") pod \"3968f1ea-e378-4dda-bbb2-d83835be0d92\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") "
Nov 03 20:34:25.395832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.395713    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\" (UniqueName: \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\") pod \"3968f1ea-e378-4dda-bbb2-d83835be0d92\" (UID: \"3968f1ea-e378-4dda-bbb2-d83835be0d92\") "
Nov 03 20:34:25.396035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.396006    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume" pod "3968f1ea-e378-4dda-bbb2-d83835be0d92" (UID: "3968f1ea-e378-4dda-bbb2-d83835be0d92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:25.396198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.396180    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f1ea-e378-4dda-bbb2-d83835be0d92-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:25.398071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.398041    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw" pod "3968f1ea-e378-4dda-bbb2-d83835be0d92" (UID: "3968f1ea-e378-4dda-bbb2-d83835be0d92"). InnerVolumeSpecName "kube-api-access-chknw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:25.498615 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.497759    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chknw\" (UniqueName: \"kubernetes.io/projected/3968f1ea-e378-4dda-bbb2-d83835be0d92-kube-api-access-chknw\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:25.958725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.958696    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"]
Nov 03 20:34:25.971269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:25.971238    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e40816-2rbkg"]
Nov 03 20:34:26.153316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.153287    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"
Nov 03 20:34:26.153517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.153318    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5" event={"ID":"3968f1ea-e378-4dda-bbb2-d83835be0d92","Type":"ContainerDied","Data":"4e7af37055906f5f9b93e714664dd4940bb74b0893a7da9cdef6cf5d50086a9e"}
Nov 03 20:34:26.153517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.153393    5424 scope.go:122] "RemoveContainer" containerID="e4f9043ef57dffb6f1e6677b8a2440fa55425f4fd2f0d45f814c20a80d63e893"
Nov 03 20:34:26.204910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.204836    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:34:26.211760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.211731    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e40816-598fd5f486-5ffp5"]
Nov 03 20:34:26.238558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.237145    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:26.238558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:26.237245    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:27.243220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:27.243126    5424 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 20:34:27.243220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:27.243201    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 20:34:27.921782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:27.921746    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" path="/var/lib/kubelet/pods/3968f1ea-e378-4dda-bbb2-d83835be0d92/volumes"
Nov 03 20:34:27.922365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:27.922342    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7905e3d5-a316-45c2-bcf0-42b43d7fa2a0" path="/var/lib/kubelet/pods/7905e3d5-a316-45c2-bcf0-42b43d7fa2a0/volumes"
Nov 03 20:34:28.229436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:28.229356    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:28.229825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:28.229804    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:28.230683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:28.230664    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:34:34.687925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:34.687888    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:34:34.689988 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:34.689930    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" containerID="cri-o://7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5" gracePeriod=30
Nov 03 20:34:35.111304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.111268    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:34:35.111782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.111768    5424 cpu_manager.go:414] "containerMap: removing container" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing"
Nov 03 20:34:35.111855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.111783    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing"
Nov 03 20:34:35.111911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.111864    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="3968f1ea-e378-4dda-bbb2-d83835be0d92" containerName="payload-processing"
Nov 03 20:34:35.116839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.116810    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.145207 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.145168    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:34:35.190693 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190654    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190719    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190766    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190790    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dr4\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190815    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190855    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.190920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190921    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.191212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190962    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.191212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.190992    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.240940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.240900    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60efdf-openshift-default-dockercfg-289b9\""
Nov 03 20:34:35.293597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293556    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293618    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293661    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293693    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293735    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293755    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dr4\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.293807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293778    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293822    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.293896    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.294130    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294308 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.294191    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294461 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.294434    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.294439    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.294695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.294623    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.296596 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.296570    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.296752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.296735    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.314671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.314645    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.315045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.315024    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dr4\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.442887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.442780    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:35.766337 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:34:35.766300    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ef0c4f_3793_4657_9450_c61cf3f934c3.slice/crio-7f5d72057b3aa953c9fae7bc78f6a130dc07f9a0228ab37086baf675adae64e8 WatchSource:0}: Error finding container 7f5d72057b3aa953c9fae7bc78f6a130dc07f9a0228ab37086baf675adae64e8: Status 404 returned error can't find the container with id 7f5d72057b3aa953c9fae7bc78f6a130dc07f9a0228ab37086baf675adae64e8
Nov 03 20:34:35.767153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.767127    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:34:35.768900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.768868    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:35.768986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.768933    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:35.768986 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:35.768961    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:36.265069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.264948    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" event={"ID":"c9ef0c4f-3793-4657-9450-c61cf3f934c3","Type":"ContainerStarted","Data":"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"}
Nov 03 20:34:36.265420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.265295    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" event={"ID":"c9ef0c4f-3793-4657-9450-c61cf3f934c3","Type":"ContainerStarted","Data":"7f5d72057b3aa953c9fae7bc78f6a130dc07f9a0228ab37086baf675adae64e8"}
Nov 03 20:34:36.312166 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.312104    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podStartSLOduration=1.312081519 podStartE2EDuration="1.312081519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:36.307717707 +0000 UTC m=+1818.516733716" watchObservedRunningTime="2026-11-03 20:34:36.312081519 +0000 UTC m=+1818.521097517"
Nov 03 20:34:36.450030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.449986    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:36.454554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.452283    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:36.454554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:36.452345    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:37.444292 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:37.444196    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:37.444928 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:37.444889    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:38.449021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:38.448973    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:38.449577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:38.449060    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:39.443982 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:39.443941    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:39.444340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:39.444296    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:39.777870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:39.777832    5424 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:34:39.778358 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:39.777898    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 20:34:40.096021 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.095990    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:34:40.185497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185455    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185575    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185692    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185738    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185785    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185853    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185886    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.185944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185920    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.186183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.185968    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\") pod \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\" (UID: \"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4\") "
Nov 03 20:34:40.186183 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186034    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:40.186443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186422    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.186573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186452    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.186573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186481    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.186740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186719    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.186853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.186837    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.189121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.189084    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:40.190317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.190276    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "kube-api-access-79jn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:40.191969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.191933    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.192262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.192234    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token" pod "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" (UID: "5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289766    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289800    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289810    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289818    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289828    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289836    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jn5\" (UniqueName: \"kubernetes.io/projected/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-kube-api-access-79jn5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289844    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.289860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.289852    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.298054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.298022    5424 generic.go:356] "Generic (PLEG): container finished" podID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerID="7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5" exitCode=0
Nov 03 20:34:40.298228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.298085    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"
Nov 03 20:34:40.298228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.298109    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" event={"ID":"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4","Type":"ContainerDied","Data":"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"}
Nov 03 20:34:40.298228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.298173    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw" event={"ID":"5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4","Type":"ContainerDied","Data":"6898270d2c98a5d63c3b2abb6482cb3ef05bb8f2d1b142dba1de0e14d5e87012"}
Nov 03 20:34:40.298228 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.298202    5424 scope.go:122] "RemoveContainer" containerID="7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"
Nov 03 20:34:40.309429 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.309260    5424 scope.go:122] "RemoveContainer" containerID="7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"
Nov 03 20:34:40.309755 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:40.309733    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5\": container with ID starting with 7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5 not found: ID does not exist" containerID="7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"
Nov 03 20:34:40.309859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.309769    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5"} err="failed to get container status \"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5\": rpc error: code = NotFound desc = could not find container \"7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5\": container with ID starting with 7cbf6325ba79cdb50f6bc12a021c82823e1f353cbb4cd17d0b53ab630f0c1cc5 not found: ID does not exist"
Nov 03 20:34:40.353653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.353328    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:34:40.355950 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.355926    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wrbkw"]
Nov 03 20:34:40.444555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.444201    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:40.444555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:40.444265    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:41.444126 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:41.444084    5424 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:34:41.444598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:41.444173    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 20:34:41.920288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:41.920258    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" path="/var/lib/kubelet/pods/5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4/volumes"
Nov 03 20:34:42.289562 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:34:42.289490    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:34:42.314846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.314803    5424 generic.go:356] "Generic (PLEG): container finished" podID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerID="241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0" exitCode=0
Nov 03 20:34:42.315055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.314851    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerDied","Data":"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"}
Nov 03 20:34:42.315326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.315308    5424 scope.go:122] "RemoveContainer" containerID="241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"
Nov 03 20:34:42.317942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.317625    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:42.317942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.317870    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:42.318115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.318077    5424 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:34:42.448657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.448628    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:42.449249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.449215    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:42.450001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.449985    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:34:42.849144 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:42.849107    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:34:43.347446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:43.347404    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerStarted","Data":"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"}
Nov 03 20:34:43.973680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:43.973648    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:34:43.975129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:43.975106    5424 patch_prober.go:29] interesting pod/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:34:43.975310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:43.975157    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:34:44.974059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:44.974014    5424 patch_prober.go:29] interesting pod/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:34:44.974506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:44.974075    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:34:45.973314 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:45.973278    5424 patch_prober.go:29] interesting pod/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:34:45.973510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:45.973349    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:34:46.973234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:46.973199    5424 patch_prober.go:29] interesting pod/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:34:46.973717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:46.973254    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:34:47.973033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:47.972995    5424 patch_prober.go:29] interesting pod/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:34:47.973265 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:47.973060    5424 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:34:48.979453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:48.979422    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:34:48.980060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:48.979949    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:34:48.980791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:34:48.980770    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:35:12.287640 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:35:12.287602    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:35:21.448257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:35:21.448224    5424 scope.go:122] "RemoveContainer" containerID="365d1dc4f41525c6af8e28f0b43ac404cc3e6b28c348ddd890cf0a8eda39f71d"
Nov 03 20:35:21.459739 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:35:21.459715    5424 scope.go:122] "RemoveContainer" containerID="cd2aee82ffce753a8f5e6190f89bb807bee6fbff9e5ecd34806ef6afb19dfb55"
Nov 03 20:35:21.468469 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:35:21.468445    5424 scope.go:122] "RemoveContainer" containerID="02f5e76335cf9722e3f009211695ba387302153c98f71c370707aa0a9a61e8fb"
Nov 03 20:35:21.477435 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:35:21.477413    5424 scope.go:122] "RemoveContainer" containerID="4cb830cbb384e4c5ace84f710d16cb783df51cbccfeda01761600ecef1265562"
Nov 03 20:35:42.296272 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:35:42.296225    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:36:01.852724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.852636    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"]
Nov 03 20:36:01.853397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.853374    5424 cpu_manager.go:414] "containerMap: removing container" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy"
Nov 03 20:36:01.853491 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.853400    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy"
Nov 03 20:36:01.853587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.853565    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="5475de1a-c6f9-4cc8-a8f5-28ab56b30ac4" containerName="istio-proxy"
Nov 03 20:36:01.855858 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.855838    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:01.871907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.871879    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:01.875948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.875920    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"]
Nov 03 20:36:01.883969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.883949    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:01.951599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.951567    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892l2\" (UniqueName: \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:01.951770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.951753    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:01.951819 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:01.951786    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.053403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.053371    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.053403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.053407    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.053658 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.053463    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892l2\" (UniqueName: \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.054109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.054085    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.055860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.055841    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.072904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.072878    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892l2\" (UniqueName: \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\") pod \"maas-api-revoke-sub-bd00d862-xwqbs\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.168369 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.168288    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:02.465491 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:36:02.465452    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc038e5be_340a_4896_a265_65c9552cdfe9.slice/crio-66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a WatchSource:0}: Error finding container 66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a: Status 404 returned error can't find the container with id 66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a
Nov 03 20:36:02.465718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.465688    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"]
Nov 03 20:36:02.922042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.922005    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs" event={"ID":"c038e5be-340a-4896-a265-65c9552cdfe9","Type":"ContainerStarted","Data":"c4e16b7ffc0817180deb37be5d20e913704fb737b69df43a4d15a7bf4fdaf91f"}
Nov 03 20:36:02.922692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.922060    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs" event={"ID":"c038e5be-340a-4896-a265-65c9552cdfe9","Type":"ContainerStarted","Data":"66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a"}
Nov 03 20:36:02.963670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:02.963616    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs" podStartSLOduration=1.963601387 podStartE2EDuration="1.963601387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:02.961335605 +0000 UTC m=+1905.170351602" watchObservedRunningTime="2026-11-03 20:36:02.963601387 +0000 UTC m=+1905.172617362"
Nov 03 20:36:03.929878 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:03.929843    5424 generic.go:356] "Generic (PLEG): container finished" podID="c038e5be-340a-4896-a265-65c9552cdfe9" containerID="c4e16b7ffc0817180deb37be5d20e913704fb737b69df43a4d15a7bf4fdaf91f" exitCode=0
Nov 03 20:36:03.930368 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:03.929929    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs" event={"ID":"c038e5be-340a-4896-a265-65c9552cdfe9","Type":"ContainerDied","Data":"c4e16b7ffc0817180deb37be5d20e913704fb737b69df43a4d15a7bf4fdaf91f"}
Nov 03 20:36:05.074298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.074271    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:05.203741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.203658    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\") pod \"c038e5be-340a-4896-a265-65c9552cdfe9\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") "
Nov 03 20:36:05.203900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.203760    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\" (UniqueName: \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\") pod \"c038e5be-340a-4896-a265-65c9552cdfe9\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") "
Nov 03 20:36:05.203943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.203900    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\") pod \"c038e5be-340a-4896-a265-65c9552cdfe9\" (UID: \"c038e5be-340a-4896-a265-65c9552cdfe9\") "
Nov 03 20:36:05.204242 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.204208    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca" pod "c038e5be-340a-4896-a265-65c9552cdfe9" (UID: "c038e5be-340a-4896-a265-65c9552cdfe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:05.204470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.204333    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:05.206065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.206034    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca" pod "c038e5be-340a-4896-a265-65c9552cdfe9" (UID: "c038e5be-340a-4896-a265-65c9552cdfe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:05.206065 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.206036    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2" pod "c038e5be-340a-4896-a265-65c9552cdfe9" (UID: "c038e5be-340a-4896-a265-65c9552cdfe9"). InnerVolumeSpecName "kube-api-access-892l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:05.305734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.305690    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c038e5be-340a-4896-a265-65c9552cdfe9-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:05.305734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.305728    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892l2\" (UniqueName: \"kubernetes.io/projected/c038e5be-340a-4896-a265-65c9552cdfe9-kube-api-access-892l2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:05.944326 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.944294    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs" event={"ID":"c038e5be-340a-4896-a265-65c9552cdfe9","Type":"ContainerDied","Data":"66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a"}
Nov 03 20:36:05.944494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.944332    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"
Nov 03 20:36:05.944494 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:05.944345    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f9b4e1ede9ebc67568384d036d2837f2dd584cba819948bab805f2858d9a7a"
Nov 03 20:36:12.285237 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:36:12.285199    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:36:13.331441 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.331404    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"]
Nov 03 20:36:13.331900 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.331886    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c038e5be-340a-4896-a265-65c9552cdfe9" containerName="revoke-keys"
Nov 03 20:36:13.331946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.331901    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c038e5be-340a-4896-a265-65c9552cdfe9" containerName="revoke-keys"
Nov 03 20:36:13.331979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.331973    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c038e5be-340a-4896-a265-65c9552cdfe9" containerName="revoke-keys"
Nov 03 20:36:13.335344 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.335325    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.350614 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.350583    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:13.352673 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.352648    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"]
Nov 03 20:36:13.362119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.362095    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:13.399699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.399665    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhl6z\" (UniqueName: \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.399839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.399721    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.399839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.399799    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.500745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.500713    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl6z\" (UniqueName: \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.500941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.500758    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.500941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.500779    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.501479 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.501457    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.503307 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.503284    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.520206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.520184    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl6z\" (UniqueName: \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\") pod \"maas-api-revoke-sub-16eb999f-qj8b8\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.649399 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.649331    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:13.952310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.952277    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"]
Nov 03 20:36:13.953293 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:36:13.953264    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb09928_62ac_403b_8683_7d01c47467a0.slice/crio-a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee WatchSource:0}: Error finding container a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee: Status 404 returned error can't find the container with id a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee
Nov 03 20:36:13.998923 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:13.998896    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8" event={"ID":"fbb09928-62ac-403b-8683-7d01c47467a0","Type":"ContainerStarted","Data":"a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee"}
Nov 03 20:36:15.007001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:15.006963    5424 generic.go:356] "Generic (PLEG): container finished" podID="fbb09928-62ac-403b-8683-7d01c47467a0" containerID="5adb45d4a76034995fd8a4f65ace5b1717f336652de10dad2c5a19267174621e" exitCode=0
Nov 03 20:36:15.007373 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:15.007051    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8" event={"ID":"fbb09928-62ac-403b-8683-7d01c47467a0","Type":"ContainerDied","Data":"5adb45d4a76034995fd8a4f65ace5b1717f336652de10dad2c5a19267174621e"}
Nov 03 20:36:16.149118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.149093    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:16.231027 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.230992    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\" (UniqueName: \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\") pod \"fbb09928-62ac-403b-8683-7d01c47467a0\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") "
Nov 03 20:36:16.231201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.231108    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\") pod \"fbb09928-62ac-403b-8683-7d01c47467a0\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") "
Nov 03 20:36:16.231201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.231192    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\") pod \"fbb09928-62ac-403b-8683-7d01c47467a0\" (UID: \"fbb09928-62ac-403b-8683-7d01c47467a0\") "
Nov 03 20:36:16.231444 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.231418    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca" pod "fbb09928-62ac-403b-8683-7d01c47467a0" (UID: "fbb09928-62ac-403b-8683-7d01c47467a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:16.233483 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.233456    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca" pod "fbb09928-62ac-403b-8683-7d01c47467a0" (UID: "fbb09928-62ac-403b-8683-7d01c47467a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:16.233627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.233495    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z" pod "fbb09928-62ac-403b-8683-7d01c47467a0" (UID: "fbb09928-62ac-403b-8683-7d01c47467a0"). InnerVolumeSpecName "kube-api-access-xhl6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:16.332204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.332125    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:16.332204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.332158    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb09928-62ac-403b-8683-7d01c47467a0-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:16.332204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:16.332167    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhl6z\" (UniqueName: \"kubernetes.io/projected/fbb09928-62ac-403b-8683-7d01c47467a0-kube-api-access-xhl6z\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:17.020051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:17.020022    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"
Nov 03 20:36:17.020051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:17.020033    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8" event={"ID":"fbb09928-62ac-403b-8683-7d01c47467a0","Type":"ContainerDied","Data":"a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee"}
Nov 03 20:36:17.020281 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:17.020077    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ca02e03eb5416d6843f3b5af81d2e068f65f343e00123f9760d969764ecdee"
Nov 03 20:36:21.523248 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:21.523213    5424 scope.go:122] "RemoveContainer" containerID="45b62d6240a3f8a64dab5baa4953ff82bdd7ac35b084cfd6250150868059005d"
Nov 03 20:36:21.533187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:21.533161    5424 scope.go:122] "RemoveContainer" containerID="01cdb5afcb3cf1e33fa9b2ceddd062b6e3eadcfbd487e169df87245fdeeb2e2d"
Nov 03 20:36:23.195957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.195920    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"]
Nov 03 20:36:23.196670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.196649    5424 cpu_manager.go:414] "containerMap: removing container" podUID="fbb09928-62ac-403b-8683-7d01c47467a0" containerName="revoke-keys"
Nov 03 20:36:23.196670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.196671    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb09928-62ac-403b-8683-7d01c47467a0" containerName="revoke-keys"
Nov 03 20:36:23.196854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.196787    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb09928-62ac-403b-8683-7d01c47467a0" containerName="revoke-keys"
Nov 03 20:36:23.201981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.201959    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.219653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.219626    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"]
Nov 03 20:36:23.227891 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.227863    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:23.228028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.227916    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:23.309493 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.309462    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fh6p\" (UniqueName: \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.309715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.309637    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.309775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.309730    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.410717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.410683    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.410868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.410746    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fh6p\" (UniqueName: \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.410868 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.410851    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.411328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.411307    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.413318 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.413299    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.429814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.429793    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fh6p\" (UniqueName: \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\") pod \"maas-api-revoke-sub-00960573-lm8b9\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.515831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.515804    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:23.823262 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:23.823228    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"]
Nov 03 20:36:23.824311 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:36:23.824267    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537d76b8_9231_49d9_be89_4a9670bfe08e.slice/crio-e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6 WatchSource:0}: Error finding container e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6: Status 404 returned error can't find the container with id e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6
Nov 03 20:36:24.068259 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:24.068223    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9" event={"ID":"537d76b8-9231-49d9-be89-4a9670bfe08e","Type":"ContainerStarted","Data":"e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6"}
Nov 03 20:36:25.076496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:25.076457    5424 generic.go:356] "Generic (PLEG): container finished" podID="537d76b8-9231-49d9-be89-4a9670bfe08e" containerID="6950a9f9ae2fcfd1eb57d367e799e2460ecd2a90cb9190886f773efdd71b9c18" exitCode=0
Nov 03 20:36:25.076901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:25.076578    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9" event={"ID":"537d76b8-9231-49d9-be89-4a9670bfe08e","Type":"ContainerDied","Data":"6950a9f9ae2fcfd1eb57d367e799e2460ecd2a90cb9190886f773efdd71b9c18"}
Nov 03 20:36:26.216767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.216741    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:26.345908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.345827    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\") pod \"537d76b8-9231-49d9-be89-4a9670bfe08e\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") "
Nov 03 20:36:26.345908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.345869    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\" (UniqueName: \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\") pod \"537d76b8-9231-49d9-be89-4a9670bfe08e\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") "
Nov 03 20:36:26.346114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.346053    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\") pod \"537d76b8-9231-49d9-be89-4a9670bfe08e\" (UID: \"537d76b8-9231-49d9-be89-4a9670bfe08e\") "
Nov 03 20:36:26.346412 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.346385    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca" pod "537d76b8-9231-49d9-be89-4a9670bfe08e" (UID: "537d76b8-9231-49d9-be89-4a9670bfe08e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:26.346593 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.346576    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:26.348115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.348087    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca" pod "537d76b8-9231-49d9-be89-4a9670bfe08e" (UID: "537d76b8-9231-49d9-be89-4a9670bfe08e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:26.348115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.348130    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p" pod "537d76b8-9231-49d9-be89-4a9670bfe08e" (UID: "537d76b8-9231-49d9-be89-4a9670bfe08e"). InnerVolumeSpecName "kube-api-access-9fh6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:26.447930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.447897    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537d76b8-9231-49d9-be89-4a9670bfe08e-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:26.447930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:26.447927    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fh6p\" (UniqueName: \"kubernetes.io/projected/537d76b8-9231-49d9-be89-4a9670bfe08e-kube-api-access-9fh6p\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:27.090320 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:27.090283    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9" event={"ID":"537d76b8-9231-49d9-be89-4a9670bfe08e","Type":"ContainerDied","Data":"e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6"}
Nov 03 20:36:27.090495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:27.090342    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e3a45f6ba24d11b497f96f02fd0f9a679bbb3ed40090a2f5c88f5c6710a5e6"
Nov 03 20:36:27.090495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:27.090293    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"
Nov 03 20:36:42.288596 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:36:42.288552    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:36:45.964402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:45.964365    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:36:45.965303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:45.965266    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy" containerID="cri-o://a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717" gracePeriod=30
Nov 03 20:36:47.028135 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.028094    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"]
Nov 03 20:36:47.040630 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.040593    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f98f9ea2-jhnhh"]
Nov 03 20:36:47.904101 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.904067    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"]
Nov 03 20:36:47.905200 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.905167    5424 cpu_manager.go:414] "containerMap: removing container" podUID="537d76b8-9231-49d9-be89-4a9670bfe08e" containerName="revoke-keys"
Nov 03 20:36:47.905401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.905383    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537d76b8-9231-49d9-be89-4a9670bfe08e" containerName="revoke-keys"
Nov 03 20:36:47.905701 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.905673    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="537d76b8-9231-49d9-be89-4a9670bfe08e" containerName="revoke-keys"
Nov 03 20:36:47.909327 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.909308    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:47.919430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.919402    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6240bd-4a29-4986-b8d4-b6158da09a5c" path="/var/lib/kubelet/pods/5d6240bd-4a29-4986-b8d4-b6158da09a5c/volumes"
Nov 03 20:36:47.926410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.926222    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"]
Nov 03 20:36:47.934751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.934728    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:47.934901 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.934771    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:47.994904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.994868    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:47.995084 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.994932    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:47.995151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:47.995123    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4r4\" (UniqueName: \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.096127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.096093    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.096715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.096212    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.096715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.096512    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4r4\" (UniqueName: \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.096943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.096913    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.099155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.099132    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.116257 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.116228    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4r4\" (UniqueName: \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\") pod \"maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.224338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.224258    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:48.531054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:48.531022    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"]
Nov 03 20:36:48.531815 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:36:48.531787    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe167e98_2aaa_4a3d_95e8_a2cb2b3eeea1.slice/crio-6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85 WatchSource:0}: Error finding container 6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85: Status 404 returned error can't find the container with id 6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85
Nov 03 20:36:49.237296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:49.237257    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm" event={"ID":"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1","Type":"ContainerStarted","Data":"7bd46e7bd61c152e17a85e8cdc971b4a7f3117e056f20bbc8020ec2a8f304d0b"}
Nov 03 20:36:49.237738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:49.237323    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm" event={"ID":"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1","Type":"ContainerStarted","Data":"6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85"}
Nov 03 20:36:49.276852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:49.276796    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm" podStartSLOduration=2.2767817470000002 podStartE2EDuration="2.276781747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:49.274755863 +0000 UTC m=+1951.483771860" watchObservedRunningTime="2026-11-03 20:36:49.276781747 +0000 UTC m=+1951.485797744"
Nov 03 20:36:50.245069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:50.245034    5424 generic.go:356] "Generic (PLEG): container finished" podID="fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" containerID="7bd46e7bd61c152e17a85e8cdc971b4a7f3117e056f20bbc8020ec2a8f304d0b" exitCode=0
Nov 03 20:36:50.245496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:50.245078    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm" event={"ID":"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1","Type":"ContainerDied","Data":"7bd46e7bd61c152e17a85e8cdc971b4a7f3117e056f20bbc8020ec2a8f304d0b"}
Nov 03 20:36:51.229035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.229006    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:36:51.254888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.254779    5424 generic.go:356] "Generic (PLEG): container finished" podID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerID="a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717" exitCode=0
Nov 03 20:36:51.254888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.254850    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"
Nov 03 20:36:51.254888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.254857    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" event={"ID":"c9ef0c4f-3793-4657-9450-c61cf3f934c3","Type":"ContainerDied","Data":"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"}
Nov 03 20:36:51.255507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.254949    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b" event={"ID":"c9ef0c4f-3793-4657-9450-c61cf3f934c3","Type":"ContainerDied","Data":"7f5d72057b3aa953c9fae7bc78f6a130dc07f9a0228ab37086baf675adae64e8"}
Nov 03 20:36:51.255507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.255011    5424 scope.go:122] "RemoveContainer" containerID="a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"
Nov 03 20:36:51.266946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.266922    5424 scope.go:122] "RemoveContainer" containerID="a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"
Nov 03 20:36:51.267378 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:36:51.267356    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717\": container with ID starting with a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717 not found: ID does not exist" containerID="a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"
Nov 03 20:36:51.267456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.267387    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717"} err="failed to get container status \"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717\": rpc error: code = NotFound desc = could not find container \"a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717\": container with ID starting with a8aa38f86bca6e40034e200cd524d918a1dd47de81dc667a7b2893d297c67717 not found: ID does not exist"
Nov 03 20:36:51.332002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.331967    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332092    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332218 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332194    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332271    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332326    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332360    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.332405 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332386    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332447    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332518 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332476    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332582    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\") pod \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\" (UID: \"c9ef0c4f-3793-4657-9450-c61cf3f934c3\") "
Nov 03 20:36:51.332725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332673    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.333180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332845    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.333180 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.332887    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:51.333347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.333176    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.333347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.333212    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.333347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.333238    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.333347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.333251    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.333347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.333264    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.335941 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.335919    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.336046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.335995    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.337404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.337378    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "kube-api-access-m7dr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.338565 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.338546    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo" pod "c9ef0c4f-3793-4657-9450-c61cf3f934c3" (UID: "c9ef0c4f-3793-4657-9450-c61cf3f934c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:51.405165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.405136    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:51.434370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.434329    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7dr4\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-kube-api-access-m7dr4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.434370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.434359    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.434370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.434374    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.434632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.434387    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.434632 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.434399    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ef0c4f-3793-4657-9450-c61cf3f934c3-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.534972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.534886    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\") pod \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") "
Nov 03 20:36:51.534972 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.534941    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\" (UniqueName: \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\") pod \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") "
Nov 03 20:36:51.535208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.534977    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\") pod \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\" (UID: \"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1\") "
Nov 03 20:36:51.535296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.535265    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca" pod "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" (UID: "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:51.535439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.535420    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.537272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.537245    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca" pod "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" (UID: "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:51.537383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.537266    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4" pod "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" (UID: "fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1"). InnerVolumeSpecName "kube-api-access-mp4r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.610004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.609967    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:36:51.612937 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.612903    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-rcq5b"]
Nov 03 20:36:51.636748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.636717    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4r4\" (UniqueName: \"kubernetes.io/projected/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-kube-api-access-mp4r4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.636748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.636746    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.919505 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:51.919425    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" path="/var/lib/kubelet/pods/c9ef0c4f-3793-4657-9450-c61cf3f934c3/volumes"
Nov 03 20:36:52.263560 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:52.263518    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"
Nov 03 20:36:52.264047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:52.263575    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm" event={"ID":"fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1","Type":"ContainerDied","Data":"6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85"}
Nov 03 20:36:52.264047 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:52.263634    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed84cc282d5a3f1a2a8b0151945a5717c7577a4ea714912581e22030548cd85"
Nov 03 20:36:57.030717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.030681    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:36:57.031185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.030958    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" podUID="13fc3847-b72e-4bcd-b8cc-739de4da5979" containerName="maas-api" containerID="cri-o://eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3" gracePeriod=30
Nov 03 20:36:57.303679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.303648    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:36:57.310167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.310135    5424 generic.go:356] "Generic (PLEG): container finished" podID="13fc3847-b72e-4bcd-b8cc-739de4da5979" containerID="eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3" exitCode=0
Nov 03 20:36:57.310337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.310185    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"
Nov 03 20:36:57.310337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.310212    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" event={"ID":"13fc3847-b72e-4bcd-b8cc-739de4da5979","Type":"ContainerDied","Data":"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"}
Nov 03 20:36:57.310337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.310290    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c" event={"ID":"13fc3847-b72e-4bcd-b8cc-739de4da5979","Type":"ContainerDied","Data":"1406989327f7e08588add7cb30cd35150ed6519e43f1b3d3dafc1691a20de9c5"}
Nov 03 20:36:57.310337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.310319    5424 scope.go:122] "RemoveContainer" containerID="eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"
Nov 03 20:36:57.320142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.320124    5424 scope.go:122] "RemoveContainer" containerID="eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"
Nov 03 20:36:57.320424 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:36:57.320402    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3\": container with ID starting with eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3 not found: ID does not exist" containerID="eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"
Nov 03 20:36:57.320475 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.320435    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3"} err="failed to get container status \"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3\": rpc error: code = NotFound desc = could not find container \"eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3\": container with ID starting with eade8f52e1b4c534651d609d1d7734d0a49b190160c2ef03837d221b2685dcd3 not found: ID does not exist"
Nov 03 20:36:57.394374 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.394350    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\") pod \"13fc3847-b72e-4bcd-b8cc-739de4da5979\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") "
Nov 03 20:36:57.394572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.394458    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\") pod \"13fc3847-b72e-4bcd-b8cc-739de4da5979\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") "
Nov 03 20:36:57.394572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.394479    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\" (UniqueName: \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\") pod \"13fc3847-b72e-4bcd-b8cc-739de4da5979\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") "
Nov 03 20:36:57.394572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.394556    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") pod \"13fc3847-b72e-4bcd-b8cc-739de4da5979\" (UID: \"13fc3847-b72e-4bcd-b8cc-739de4da5979\") "
Nov 03 20:36:57.394822 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.394797    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca" pod "13fc3847-b72e-4bcd-b8cc-739de4da5979" (UID: "13fc3847-b72e-4bcd-b8cc-739de4da5979"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:57.396713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.396682    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs" pod "13fc3847-b72e-4bcd-b8cc-739de4da5979" (UID: "13fc3847-b72e-4bcd-b8cc-739de4da5979"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:57.396814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.396736    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls" pod "13fc3847-b72e-4bcd-b8cc-739de4da5979" (UID: "13fc3847-b72e-4bcd-b8cc-739de4da5979"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:57.396814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.396739    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr" pod "13fc3847-b72e-4bcd-b8cc-739de4da5979" (UID: "13fc3847-b72e-4bcd-b8cc-739de4da5979"). InnerVolumeSpecName "kube-api-access-zxmgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:57.502610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.495798    5424 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-metrics-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:57.502610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.495833    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmgr\" (UniqueName: \"kubernetes.io/projected/13fc3847-b72e-4bcd-b8cc-739de4da5979-kube-api-access-zxmgr\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:57.502610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.495848    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13fc3847-b72e-4bcd-b8cc-739de4da5979-maas-api-tls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:57.502610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.495862    5424 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13fc3847-b72e-4bcd-b8cc-739de4da5979-openshift-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:36:57.655766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.655727    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:36:57.658643 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.658616    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1d159a-757c897574-5m94c"]
Nov 03 20:36:57.919558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:36:57.919449    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fc3847-b72e-4bcd-b8cc-739de4da5979" path="/var/lib/kubelet/pods/13fc3847-b72e-4bcd-b8cc-739de4da5979/volumes"
Nov 03 20:37:04.639888 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:04.639843    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"]
Nov 03 20:37:04.652365 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:04.652337    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9ba83-5rvqm"]
Nov 03 20:37:05.918443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:05.918408    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" path="/var/lib/kubelet/pods/fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1/volumes"
Nov 03 20:37:05.972329 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:05.972293    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:37:05.972865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:05.972809    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy" containerID="cri-o://f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016" gracePeriod=30
Nov 03 20:37:11.242519 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.242462    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:37:11.359296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359262    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359337    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359371    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359389    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359423    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359464 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359446    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359467    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359515    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359565    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\") pod \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\" (UID: \"34081f6f-69f1-4a88-a2df-06bc6b0e2c72\") "
Nov 03 20:37:11.359764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359673    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.359764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359689    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:11.360025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359833    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.360025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.359847    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.360025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.360000    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.360154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.360063    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.360154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.360083    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.360154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.360095    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.360154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.360107    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.361897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.361871    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.362029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.361892    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:11.362376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.362344    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:11.362376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.362361    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2" pod "34081f6f-69f1-4a88-a2df-06bc6b0e2c72" (UID: "34081f6f-69f1-4a88-a2df-06bc6b0e2c72"). InnerVolumeSpecName "kube-api-access-d8sv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:11.421764 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.421709    5424 generic.go:356] "Generic (PLEG): container finished" podID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerID="f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016" exitCode=0
Nov 03 20:37:11.421985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.421790    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"
Nov 03 20:37:11.421985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.421824    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" event={"ID":"34081f6f-69f1-4a88-a2df-06bc6b0e2c72","Type":"ContainerDied","Data":"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"}
Nov 03 20:37:11.421985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.421876    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn" event={"ID":"34081f6f-69f1-4a88-a2df-06bc6b0e2c72","Type":"ContainerDied","Data":"06c4a6ac66bbd641de3f589afdca507d37bc0259b8942c13a0136445609749d5"}
Nov 03 20:37:11.421985 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.421933    5424 scope.go:122] "RemoveContainer" containerID="f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"
Nov 03 20:37:11.453015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.446205    5424 scope.go:122] "RemoveContainer" containerID="f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"
Nov 03 20:37:11.453015 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:11.446931    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016\": container with ID starting with f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016 not found: ID does not exist" containerID="f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"
Nov 03 20:37:11.453015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.446970    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016"} err="failed to get container status \"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016\": rpc error: code = NotFound desc = could not find container \"f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016\": container with ID starting with f979982f9ef09830b6ef5b6c1ae3eac09a94a3f0133539fcaebbf17e36e42016 not found: ID does not exist"
Nov 03 20:37:11.461237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.461203    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.461237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.461225    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.461237 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.461235    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8sv2\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-kube-api-access-d8sv2\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.461478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.461247    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.461478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.461256    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34081f6f-69f1-4a88-a2df-06bc6b0e2c72-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.482974 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.482950    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:37:11.497460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.497396    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-4b7sn"]
Nov 03 20:37:11.919907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:11.919833    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" path="/var/lib/kubelet/pods/34081f6f-69f1-4a88-a2df-06bc6b0e2c72/volumes"
Nov 03 20:37:12.284599 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:12.284564    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:37:21.562544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:21.562494    5424 scope.go:122] "RemoveContainer" containerID="0465536022db2939cc8e0764c976320f0ab074881d6a1f297730a2177606d31e"
Nov 03 20:37:21.572071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:21.572048    5424 scope.go:122] "RemoveContainer" containerID="27eac1483c175a5023d3f4c171433996698f1b1310b3624b8a77bb8c874c2627"
Nov 03 20:37:25.376376 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:25.376339    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:37:25.376883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:25.376612    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy" containerID="cri-o://3819ff3a32a94469de70be4b437abc762d1b197b83b2921ca5d8d9205f09c325" gracePeriod=30
Nov 03 20:37:30.547680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.547643    5424 generic.go:356] "Generic (PLEG): container finished" podID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerID="3819ff3a32a94469de70be4b437abc762d1b197b83b2921ca5d8d9205f09c325" exitCode=0
Nov 03 20:37:30.548075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.547717    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerDied","Data":"3819ff3a32a94469de70be4b437abc762d1b197b83b2921ca5d8d9205f09c325"}
Nov 03 20:37:30.548075 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.547808    5424 scope.go:122] "RemoveContainer" containerID="39a92da73e366ef720ee10eeaf70de9fa0ef59c5a73907d98b09425483d4aa2d"
Nov 03 20:37:30.632999 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.632969    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:37:30.660853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.660793    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.660853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.660830    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.660880    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.660925    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661001    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661046    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661081    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661204    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661273    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\") pod \"fe409dc7-7754-40d7-994a-5e92275bef6f\" (UID: \"fe409dc7-7754-40d7-994a-5e92275bef6f\") "
Nov 03 20:37:30.661400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661342    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:30.661870 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661653    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:30.661995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661782    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:30.662061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.661983    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:30.662061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.662021    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.662061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.662046    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.662258 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.662052    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:30.663421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.663395    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:30.663510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.663433    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:30.663510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.663446    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "kube-api-access-nmzlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:30.663510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.663453    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy" pod "fe409dc7-7754-40d7-994a-5e92275bef6f" (UID: "fe409dc7-7754-40d7-994a-5e92275bef6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:30.763005 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.762968    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763010    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763027    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzlb\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-kube-api-access-nmzlb\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763042    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763057    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763070    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe409dc7-7754-40d7-994a-5e92275bef6f-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:30.763182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:30.763084    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe409dc7-7754-40d7-994a-5e92275bef6f-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:31.564551 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.564513    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"
Nov 03 20:37:31.565046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.564511    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr" event={"ID":"fe409dc7-7754-40d7-994a-5e92275bef6f","Type":"ContainerDied","Data":"e4b2b7348dc8708b8c705cb5c17972051b9f1368fb598f998e1a1bdd5d0e7e70"}
Nov 03 20:37:31.565046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.564607    5424 scope.go:122] "RemoveContainer" containerID="3819ff3a32a94469de70be4b437abc762d1b197b83b2921ca5d8d9205f09c325"
Nov 03 20:37:31.627967 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.627932    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:37:31.640496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.640468    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-503c4e-openshift-default-75b698fcbc-w54nr"]
Nov 03 20:37:31.918699 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:31.918620    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" path="/var/lib/kubelet/pods/fe409dc7-7754-40d7-994a-5e92275bef6f/volumes"
Nov 03 20:37:36.373679 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:36.373646    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:37:36.423299 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:36.423266    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:37:38.092904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:38.092874    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:37:38.093325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:38.093117    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy" containerID="cri-o://4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3" gracePeriod=30
Nov 03 20:37:41.374508 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.374466    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" podUID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" containerName="payload-processing" containerID="cri-o://1d25c64e3c238c13b9a4de64ad6c4d937113dffbd518c32bd9319c1d7425a3fb" gracePeriod=25
Nov 03 20:37:41.424083 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.424035    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" podUID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" containerName="payload-pre-processing" containerID="cri-o://ac486fdfc98117f7896bf7e3b9c440be2269a83305bd6c8df8d92b352c69fc41" gracePeriod=25
Nov 03 20:37:41.653002 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.652964    5424 generic.go:356] "Generic (PLEG): container finished" podID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" containerID="1d25c64e3c238c13b9a4de64ad6c4d937113dffbd518c32bd9319c1d7425a3fb" exitCode=0
Nov 03 20:37:41.653217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.653191    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" event={"ID":"7c028d5d-bdea-479d-a0cf-2e97b902dd46","Type":"ContainerDied","Data":"1d25c64e3c238c13b9a4de64ad6c4d937113dffbd518c32bd9319c1d7425a3fb"}
Nov 03 20:37:41.656503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.656462    5424 generic.go:356] "Generic (PLEG): container finished" podID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" containerID="ac486fdfc98117f7896bf7e3b9c440be2269a83305bd6c8df8d92b352c69fc41" exitCode=0
Nov 03 20:37:41.656713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.656622    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" event={"ID":"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43","Type":"ContainerDied","Data":"ac486fdfc98117f7896bf7e3b9c440be2269a83305bd6c8df8d92b352c69fc41"}
Nov 03 20:37:41.685963 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.685934    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:37:41.711782 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.711752    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:37:41.780123 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780084    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\") pod \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") "
Nov 03 20:37:41.780310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780193    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\" (UniqueName: \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\") pod \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\" (UID: \"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43\") "
Nov 03 20:37:41.780310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780234    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\") pod \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") "
Nov 03 20:37:41.780446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780326    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\" (UniqueName: \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\") pod \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\" (UID: \"7c028d5d-bdea-479d-a0cf-2e97b902dd46\") "
Nov 03 20:37:41.780681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780491    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume" pod "f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" (UID: "f3038f1e-ef26-48c6-8f16-22a1fbcbfb43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:41.780793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780734    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume" pod "7c028d5d-bdea-479d-a0cf-2e97b902dd46" (UID: "7c028d5d-bdea-479d-a0cf-2e97b902dd46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:41.780859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780823    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:41.780859 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.780849    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c028d5d-bdea-479d-a0cf-2e97b902dd46-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:41.782815 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.782789    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l" pod "7c028d5d-bdea-479d-a0cf-2e97b902dd46" (UID: "7c028d5d-bdea-479d-a0cf-2e97b902dd46"). InnerVolumeSpecName "kube-api-access-zqw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:41.783136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.783109    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk" pod "f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" (UID: "f3038f1e-ef26-48c6-8f16-22a1fbcbfb43"). InnerVolumeSpecName "kube-api-access-2bswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:41.882389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.882284    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bswk\" (UniqueName: \"kubernetes.io/projected/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43-kube-api-access-2bswk\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:41.882389 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:41.882324    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqw8l\" (UniqueName: \"kubernetes.io/projected/7c028d5d-bdea-479d-a0cf-2e97b902dd46-kube-api-access-zqw8l\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:42.286976 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:42.286936    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:37:42.663842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.663761    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"
Nov 03 20:37:42.663842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.663804    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq" event={"ID":"f3038f1e-ef26-48c6-8f16-22a1fbcbfb43","Type":"ContainerDied","Data":"f15a5fa260b5d57f04f41a85bdac6f292eaa7ac8d46f882411aa6ebdb31a66a2"}
Nov 03 20:37:42.664459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.663871    5424 scope.go:122] "RemoveContainer" containerID="ac486fdfc98117f7896bf7e3b9c440be2269a83305bd6c8df8d92b352c69fc41"
Nov 03 20:37:42.666024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.665971    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"
Nov 03 20:37:42.666230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.666017    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v" event={"ID":"7c028d5d-bdea-479d-a0cf-2e97b902dd46","Type":"ContainerDied","Data":"bdd5eb6cf3761ad12759a287a798a041010f2099d487b6fb403bb0a84daa61a0"}
Nov 03 20:37:42.674278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.674259    5424 scope.go:122] "RemoveContainer" containerID="1d25c64e3c238c13b9a4de64ad6c4d937113dffbd518c32bd9319c1d7425a3fb"
Nov 03 20:37:42.716208 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.716171    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:37:42.729334 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.729302    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a0866c-7794b96b7-46lsq"]
Nov 03 20:37:42.762872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.762835    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:37:42.768001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:42.767957    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a0866c-5b69887578-nhf2v"]
Nov 03 20:37:43.355855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.355825    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:37:43.498236 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498202    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498246    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498305    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498424 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498372    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498460    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498514    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498588 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498573    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498574    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:43.498743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498608    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498743 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498636    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\") pod \"eb749d69-3bdb-4edf-8a8e-971697c5d109\" (UID: \"eb749d69-3bdb-4edf-8a8e-971697c5d109\") "
Nov 03 20:37:43.498918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498759    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:43.498918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498823    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:43.498918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498827    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:43.499057 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.498978    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:43.499121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.499112    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.499184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.499127    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.499184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.499136    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.499184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.499144    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.499184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.499151    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb749d69-3bdb-4edf-8a8e-971697c5d109-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.500827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.500803    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:43.501055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.501027    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:43.501332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.501309    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "kube-api-access-wqwt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:43.501332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.501321    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo" pod "eb749d69-3bdb-4edf-8a8e-971697c5d109" (UID: "eb749d69-3bdb-4edf-8a8e-971697c5d109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:43.599932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.599890    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwt9\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-kube-api-access-wqwt9\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.599932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.599925    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.599932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.599935    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.599932 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.599943    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb749d69-3bdb-4edf-8a8e-971697c5d109-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.674253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.674214    5424 generic.go:356] "Generic (PLEG): container finished" podID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerID="4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3" exitCode=0
Nov 03 20:37:43.674793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.674275    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"
Nov 03 20:37:43.674793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.674294    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerDied","Data":"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"}
Nov 03 20:37:43.674793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.674352    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6" event={"ID":"eb749d69-3bdb-4edf-8a8e-971697c5d109","Type":"ContainerDied","Data":"9f92e505ac7687bf25fd1d9a0cafbad9e99d3ac30a570d255ef82d2732c4003f"}
Nov 03 20:37:43.674793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.674403    5424 scope.go:122] "RemoveContainer" containerID="4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"
Nov 03 20:37:43.685203 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.685185    5424 scope.go:122] "RemoveContainer" containerID="241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"
Nov 03 20:37:43.694667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.694648    5424 scope.go:122] "RemoveContainer" containerID="4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"
Nov 03 20:37:43.694948 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:43.694927    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3\": container with ID starting with 4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3 not found: ID does not exist" containerID="4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"
Nov 03 20:37:43.695033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.694961    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3"} err="failed to get container status \"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3\": rpc error: code = NotFound desc = could not find container \"4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3\": container with ID starting with 4d8715ef6f05facd20f2c67535f13d792294056aa0f2a59a4cb6420549debfc3 not found: ID does not exist"
Nov 03 20:37:43.695033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.694985    5424 scope.go:122] "RemoveContainer" containerID="241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"
Nov 03 20:37:43.695230 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:43.695213    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0\": container with ID starting with 241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0 not found: ID does not exist" containerID="241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"
Nov 03 20:37:43.695274 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.695236    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0"} err="failed to get container status \"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0\": rpc error: code = NotFound desc = could not find container \"241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0\": container with ID starting with 241cf5050a25cd4d8e5839bda79de40099ab48af3265a17892e3f2ed4e4aaea0 not found: ID does not exist"
Nov 03 20:37:43.726709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.726677    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:37:43.739007 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.738977    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4b95f3-openshift-default-646b9c7fcf-mdtc6"]
Nov 03 20:37:43.919598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.919482    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" path="/var/lib/kubelet/pods/7c028d5d-bdea-479d-a0cf-2e97b902dd46/volumes"
Nov 03 20:37:43.919987 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.919972    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" path="/var/lib/kubelet/pods/eb749d69-3bdb-4edf-8a8e-971697c5d109/volumes"
Nov 03 20:37:43.920315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:43.920302    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" path="/var/lib/kubelet/pods/f3038f1e-ef26-48c6-8f16-22a1fbcbfb43/volumes"
Nov 03 20:37:49.096091 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.096049    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:37:49.127688 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.127649    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:37:49.436445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.436357    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:37:49.436622 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.436575    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" podUID="210a4013-7945-4fd5-81e3-6c0168af8b38" containerName="maas-api" containerID="cri-o://2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282" gracePeriod=30
Nov 03 20:37:49.702328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.702270    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:37:49.718226 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.718189    5424 generic.go:356] "Generic (PLEG): container finished" podID="210a4013-7945-4fd5-81e3-6c0168af8b38" containerID="2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282" exitCode=0
Nov 03 20:37:49.718419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.718249    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"
Nov 03 20:37:49.718419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.718269    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" event={"ID":"210a4013-7945-4fd5-81e3-6c0168af8b38","Type":"ContainerDied","Data":"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"}
Nov 03 20:37:49.718419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.718329    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69" event={"ID":"210a4013-7945-4fd5-81e3-6c0168af8b38","Type":"ContainerDied","Data":"85739f6066899952243b3e8d239bf4132ac77190d1ecf59e752065ac41ad2993"}
Nov 03 20:37:49.718419 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.718363    5424 scope.go:122] "RemoveContainer" containerID="2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"
Nov 03 20:37:49.728887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.728864    5424 scope.go:122] "RemoveContainer" containerID="2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"
Nov 03 20:37:49.729241 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:49.729219    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282\": container with ID starting with 2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282 not found: ID does not exist" containerID="2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"
Nov 03 20:37:49.729372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.729250    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282"} err="failed to get container status \"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282\": rpc error: code = NotFound desc = could not find container \"2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282\": container with ID starting with 2367d84ea7bdb70315ed4ed1c2a02fd158b3f45627074903149cef07721ba282 not found: ID does not exist"
Nov 03 20:37:49.765199 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.765171    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\" (UniqueName: \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\") pod \"210a4013-7945-4fd5-81e3-6c0168af8b38\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") "
Nov 03 20:37:49.765372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.765215    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\") pod \"210a4013-7945-4fd5-81e3-6c0168af8b38\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") "
Nov 03 20:37:49.765442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.765394    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\") pod \"210a4013-7945-4fd5-81e3-6c0168af8b38\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") "
Nov 03 20:37:49.765507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.765474    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\") pod \"210a4013-7945-4fd5-81e3-6c0168af8b38\" (UID: \"210a4013-7945-4fd5-81e3-6c0168af8b38\") "
Nov 03 20:37:49.765621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.765600    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca" pod "210a4013-7945-4fd5-81e3-6c0168af8b38" (UID: "210a4013-7945-4fd5-81e3-6c0168af8b38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:49.766165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.766059    5424 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/210a4013-7945-4fd5-81e3-6c0168af8b38-openshift-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:49.767635 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.767612    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs" pod "210a4013-7945-4fd5-81e3-6c0168af8b38" (UID: "210a4013-7945-4fd5-81e3-6c0168af8b38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:49.768114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.768091    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls" pod "210a4013-7945-4fd5-81e3-6c0168af8b38" (UID: "210a4013-7945-4fd5-81e3-6c0168af8b38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:49.768114 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.768105    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls" pod "210a4013-7945-4fd5-81e3-6c0168af8b38" (UID: "210a4013-7945-4fd5-81e3-6c0168af8b38"). InnerVolumeSpecName "kube-api-access-25hls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:49.866981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.866948    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hls\" (UniqueName: \"kubernetes.io/projected/210a4013-7945-4fd5-81e3-6c0168af8b38-kube-api-access-25hls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:49.866981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.866977    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-maas-api-tls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:49.866981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:49.866988    5424 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/210a4013-7945-4fd5-81e3-6c0168af8b38-metrics-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:50.057029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:50.056992    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:37:50.069725 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:50.069691    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-76b894-7cfd76969-9lp69"]
Nov 03 20:37:51.918836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:51.918804    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210a4013-7945-4fd5-81e3-6c0168af8b38" path="/var/lib/kubelet/pods/210a4013-7945-4fd5-81e3-6c0168af8b38/volumes"
Nov 03 20:37:52.550751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.550718    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"]
Nov 03 20:37:52.551167 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551152    5424 cpu_manager.go:414] "containerMap: removing container" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551169    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551184    5424 cpu_manager.go:414] "containerMap: removing container" podUID="13fc3847-b72e-4bcd-b8cc-739de4da5979" containerName="maas-api"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551190    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fc3847-b72e-4bcd-b8cc-739de4da5979" containerName="maas-api"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551204    5424 cpu_manager.go:414] "containerMap: removing container" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551210    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551217    5424 cpu_manager.go:414] "containerMap: removing container" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551222    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551229    5424 cpu_manager.go:414] "containerMap: removing container" podUID="210a4013-7945-4fd5-81e3-6c0168af8b38" containerName="maas-api"
Nov 03 20:37:52.551234 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551233    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210a4013-7945-4fd5-81e3-6c0168af8b38" containerName="maas-api"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551246    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551250    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551257    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" containerName="payload-pre-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551262    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" containerName="payload-pre-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551273    5424 cpu_manager.go:414] "containerMap: removing container" podUID="fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" containerName="revoke-keys"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551277    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" containerName="revoke-keys"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551282    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" containerName="payload-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551286    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" containerName="payload-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551365    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ef0c4f-3793-4657-9450-c61cf3f934c3" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551374    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3038f1e-ef26-48c6-8f16-22a1fbcbfb43" containerName="payload-pre-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551381    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe167e98-2aaa-4a3d-95e8-a2cb2b3eeea1" containerName="revoke-keys"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551387    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c028d5d-bdea-479d-a0cf-2e97b902dd46" containerName="payload-processing"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551392    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551398    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551405    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551411    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fc3847-b72e-4bcd-b8cc-739de4da5979" containerName="maas-api"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551418    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551423    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="34081f6f-69f1-4a88-a2df-06bc6b0e2c72" containerName="istio-proxy"
Nov 03 20:37:52.551601 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.551428    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="210a4013-7945-4fd5-81e3-6c0168af8b38" containerName="maas-api"
Nov 03 20:37:52.555562 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.555541    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.574416 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.574387    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"]
Nov 03 20:37:52.583266 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.583238    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:37:52.583400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.583269    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:37:52.583400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.583347    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:52.583400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.583397    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:52.697717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.697680    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.697879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.697736    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7pv\" (UniqueName: \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.697879 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.697806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.799096 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.799063    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.799277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.799114    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7pv\" (UniqueName: \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.799277 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.799136    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.799751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.799725    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.801657 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.801607    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.818745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.818718    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7pv\" (UniqueName: \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:52.869132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:52.869105    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:53.167838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:53.167809    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"]
Nov 03 20:37:53.169847 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:37:53.169818    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7101469a_e5bd_4e70_92d5_ec752d8192ed.slice/crio-783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c WatchSource:0}: Error finding container 783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c: Status 404 returned error can't find the container with id 783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c
Nov 03 20:37:53.756034 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:53.755994    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h" event={"ID":"7101469a-e5bd-4e70-92d5-ec752d8192ed","Type":"ContainerStarted","Data":"429d1ae092cb46921b5a52bf90682f4adecba116f54692644c3674bcc2da6e65"}
Nov 03 20:37:53.756273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:53.756080    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h" event={"ID":"7101469a-e5bd-4e70-92d5-ec752d8192ed","Type":"ContainerStarted","Data":"783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c"}
Nov 03 20:37:53.798195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:53.798145    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h" podStartSLOduration=1.798132372 podStartE2EDuration="1.798132372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:53.795426041 +0000 UTC m=+2016.004442062" watchObservedRunningTime="2026-11-03 20:37:53.798132372 +0000 UTC m=+2016.007148369"
Nov 03 20:37:54.097010 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.096926    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" podUID="9297e4fc-238d-43a5-b646-9c8688c55035" containerName="payload-processing" containerID="cri-o://e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff" gracePeriod=25
Nov 03 20:37:54.128668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.128622    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" podUID="d295797f-5a82-4283-a555-fe49bbdf4947" containerName="payload-pre-processing" containerID="cri-o://5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355" gracePeriod=25
Nov 03 20:37:54.379600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.379569    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:37:54.402920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.402894    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:37:54.517098 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517069    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\" (UniqueName: \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\") pod \"d295797f-5a82-4283-a555-fe49bbdf4947\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") "
Nov 03 20:37:54.517278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517109    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\" (UniqueName: \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\") pod \"9297e4fc-238d-43a5-b646-9c8688c55035\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") "
Nov 03 20:37:54.517278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517129    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\") pod \"d295797f-5a82-4283-a555-fe49bbdf4947\" (UID: \"d295797f-5a82-4283-a555-fe49bbdf4947\") "
Nov 03 20:37:54.517278 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517144    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\") pod \"9297e4fc-238d-43a5-b646-9c8688c55035\" (UID: \"9297e4fc-238d-43a5-b646-9c8688c55035\") "
Nov 03 20:37:54.517598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517569    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume" pod "d295797f-5a82-4283-a555-fe49bbdf4947" (UID: "d295797f-5a82-4283-a555-fe49bbdf4947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:54.517598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.517586    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume" pod "9297e4fc-238d-43a5-b646-9c8688c55035" (UID: "9297e4fc-238d-43a5-b646-9c8688c55035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:54.519663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.519510    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v" pod "d295797f-5a82-4283-a555-fe49bbdf4947" (UID: "d295797f-5a82-4283-a555-fe49bbdf4947"). InnerVolumeSpecName "kube-api-access-8mm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:54.519663 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.519576    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz" pod "9297e4fc-238d-43a5-b646-9c8688c55035" (UID: "9297e4fc-238d-43a5-b646-9c8688c55035"). InnerVolumeSpecName "kube-api-access-72hxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:54.618241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.618173    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mm9v\" (UniqueName: \"kubernetes.io/projected/d295797f-5a82-4283-a555-fe49bbdf4947-kube-api-access-8mm9v\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:54.618241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.618198    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hxz\" (UniqueName: \"kubernetes.io/projected/9297e4fc-238d-43a5-b646-9c8688c55035-kube-api-access-72hxz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:54.618241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.618208    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d295797f-5a82-4283-a555-fe49bbdf4947-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:54.618241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.618217    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9297e4fc-238d-43a5-b646-9c8688c55035-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:54.764716 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.764673    5424 generic.go:356] "Generic (PLEG): container finished" podID="7101469a-e5bd-4e70-92d5-ec752d8192ed" containerID="429d1ae092cb46921b5a52bf90682f4adecba116f54692644c3674bcc2da6e65" exitCode=0
Nov 03 20:37:54.764911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.764765    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h" event={"ID":"7101469a-e5bd-4e70-92d5-ec752d8192ed","Type":"ContainerDied","Data":"429d1ae092cb46921b5a52bf90682f4adecba116f54692644c3674bcc2da6e65"}
Nov 03 20:37:54.766042 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.766021    5424 generic.go:356] "Generic (PLEG): container finished" podID="d295797f-5a82-4283-a555-fe49bbdf4947" containerID="5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355" exitCode=0
Nov 03 20:37:54.766173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.766073    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"
Nov 03 20:37:54.766173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.766107    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" event={"ID":"d295797f-5a82-4283-a555-fe49bbdf4947","Type":"ContainerDied","Data":"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"}
Nov 03 20:37:54.766173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.766158    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h" event={"ID":"d295797f-5a82-4283-a555-fe49bbdf4947","Type":"ContainerDied","Data":"b3fe00ecff8545875ca0619581cefda96850b3bbb6dad76d75760831046d5409"}
Nov 03 20:37:54.766354 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.766186    5424 scope.go:122] "RemoveContainer" containerID="5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"
Nov 03 20:37:54.767607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.767584    5424 generic.go:356] "Generic (PLEG): container finished" podID="9297e4fc-238d-43a5-b646-9c8688c55035" containerID="e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff" exitCode=0
Nov 03 20:37:54.767721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.767655    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"
Nov 03 20:37:54.767721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.767663    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" event={"ID":"9297e4fc-238d-43a5-b646-9c8688c55035","Type":"ContainerDied","Data":"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"}
Nov 03 20:37:54.767721 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.767715    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4" event={"ID":"9297e4fc-238d-43a5-b646-9c8688c55035","Type":"ContainerDied","Data":"fe66f10fcd68e91be3d4862d65f2a81b8155f2304d8f84bb5f3501a53bf654dc"}
Nov 03 20:37:54.776120 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.776095    5424 scope.go:122] "RemoveContainer" containerID="5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"
Nov 03 20:37:54.776460 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:54.776394    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355\": container with ID starting with 5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355 not found: ID does not exist" containerID="5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"
Nov 03 20:37:54.776460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.776421    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355"} err="failed to get container status \"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355\": rpc error: code = NotFound desc = could not find container \"5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355\": container with ID starting with 5de5662434886ee32b28e99a2d38426094f4f9e6bdddbdff6cd1acbf000a3355 not found: ID does not exist"
Nov 03 20:37:54.776460 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.776438    5424 scope.go:122] "RemoveContainer" containerID="e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"
Nov 03 20:37:54.785799 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.785782    5424 scope.go:122] "RemoveContainer" containerID="e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"
Nov 03 20:37:54.786064 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:37:54.786048    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff\": container with ID starting with e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff not found: ID does not exist" containerID="e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"
Nov 03 20:37:54.786115 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.786072    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff"} err="failed to get container status \"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff\": rpc error: code = NotFound desc = could not find container \"e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff\": container with ID starting with e0452b58df05511a9b364f3d044ff0bed9aa797b76740088a8eb8ff53a323dff not found: ID does not exist"
Nov 03 20:37:54.843540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.843490    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:37:54.849013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.848984    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-76b894-6496466475-wdp5h"]
Nov 03 20:37:54.883459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.883392    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:37:54.887177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:54.887151    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-76b894-754bf8fb4b-kl6z4"]
Nov 03 20:37:55.919557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:55.919506    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9297e4fc-238d-43a5-b646-9c8688c55035" path="/var/lib/kubelet/pods/9297e4fc-238d-43a5-b646-9c8688c55035/volumes"
Nov 03 20:37:55.919994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:55.919978    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d295797f-5a82-4283-a555-fe49bbdf4947" path="/var/lib/kubelet/pods/d295797f-5a82-4283-a555-fe49bbdf4947/volumes"
Nov 03 20:37:55.928187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:55.928161    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:56.032403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.032316    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\" (UniqueName: \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\") pod \"7101469a-e5bd-4e70-92d5-ec752d8192ed\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") "
Nov 03 20:37:56.032403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.032384    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\") pod \"7101469a-e5bd-4e70-92d5-ec752d8192ed\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") "
Nov 03 20:37:56.032645 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.032444    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\") pod \"7101469a-e5bd-4e70-92d5-ec752d8192ed\" (UID: \"7101469a-e5bd-4e70-92d5-ec752d8192ed\") "
Nov 03 20:37:56.032775 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.032750    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca" pod "7101469a-e5bd-4e70-92d5-ec752d8192ed" (UID: "7101469a-e5bd-4e70-92d5-ec752d8192ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:56.034883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.034846    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca" pod "7101469a-e5bd-4e70-92d5-ec752d8192ed" (UID: "7101469a-e5bd-4e70-92d5-ec752d8192ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:56.034883 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.034872    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv" pod "7101469a-e5bd-4e70-92d5-ec752d8192ed" (UID: "7101469a-e5bd-4e70-92d5-ec752d8192ed"). InnerVolumeSpecName "kube-api-access-tq7pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:56.133650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.133607    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7pv\" (UniqueName: \"kubernetes.io/projected/7101469a-e5bd-4e70-92d5-ec752d8192ed-kube-api-access-tq7pv\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:56.133650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.133649    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:56.133650 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.133664    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7101469a-e5bd-4e70-92d5-ec752d8192ed-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:37:56.784003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.783957    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"
Nov 03 20:37:56.784003 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.783993    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h" event={"ID":"7101469a-e5bd-4e70-92d5-ec752d8192ed","Type":"ContainerDied","Data":"783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c"}
Nov 03 20:37:56.784315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:56.784046    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="783d0696fe8cd90710a366a12afc6f31e2aa50859b5dfacbfe233fcc23f6ae6c"
Nov 03 20:37:57.303816 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:57.303766    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:37:57.327153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:57.327117    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:37:58.417920 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.417885    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"]
Nov 03 20:37:58.418443 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418426    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7101469a-e5bd-4e70-92d5-ec752d8192ed" containerName="revoke-keys"
Nov 03 20:37:58.418488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418447    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7101469a-e5bd-4e70-92d5-ec752d8192ed" containerName="revoke-keys"
Nov 03 20:37:58.418488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418463    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d295797f-5a82-4283-a555-fe49bbdf4947" containerName="payload-pre-processing"
Nov 03 20:37:58.418488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418471    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295797f-5a82-4283-a555-fe49bbdf4947" containerName="payload-pre-processing"
Nov 03 20:37:58.418488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418486    5424 cpu_manager.go:414] "containerMap: removing container" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:58.418646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418495    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe409dc7-7754-40d7-994a-5e92275bef6f" containerName="istio-proxy"
Nov 03 20:37:58.418646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418505    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9297e4fc-238d-43a5-b646-9c8688c55035" containerName="payload-processing"
Nov 03 20:37:58.418646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418513    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9297e4fc-238d-43a5-b646-9c8688c55035" containerName="payload-processing"
Nov 03 20:37:58.418646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418582    5424 cpu_manager.go:414] "containerMap: removing container" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:58.418646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418589    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb749d69-3bdb-4edf-8a8e-971697c5d109" containerName="istio-proxy"
Nov 03 20:37:58.418801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418671    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7101469a-e5bd-4e70-92d5-ec752d8192ed" containerName="revoke-keys"
Nov 03 20:37:58.418801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418680    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295797f-5a82-4283-a555-fe49bbdf4947" containerName="payload-pre-processing"
Nov 03 20:37:58.418801 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.418685    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9297e4fc-238d-43a5-b646-9c8688c55035" containerName="payload-processing"
Nov 03 20:37:58.421981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.421961    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.440179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.440152    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"]
Nov 03 20:37:58.450008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.449968    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:37:58.450179 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.450020    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:37:58.459825 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.459803    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:58.459975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.459896    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:58.559785 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.559739    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.560063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.559868    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9htm\" (UniqueName: \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.560063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.559918    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.661013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.660979    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.661201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.661074    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9htm\" (UniqueName: \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.661201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.661114    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.661872 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.661799    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.663553 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.663518    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.680555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.680465    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9htm\" (UniqueName: \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:58.736151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:58.736112    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:37:59.037580 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:59.037550    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"]
Nov 03 20:37:59.038765 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:37:59.038736    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a6dee1_9af4_445d_b5b3_15f655ee0c57.slice/crio-80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333 WatchSource:0}: Error finding container 80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333: Status 404 returned error can't find the container with id 80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333
Nov 03 20:37:59.807137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:59.807100    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg" event={"ID":"b0a6dee1-9af4-445d-b5b3-15f655ee0c57","Type":"ContainerStarted","Data":"8318e6ae2b346674a3f810aa79f133de7b04fe922cabd395b5bebd150369726f"}
Nov 03 20:37:59.807520 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:59.807157    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg" event={"ID":"b0a6dee1-9af4-445d-b5b3-15f655ee0c57","Type":"ContainerStarted","Data":"80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333"}
Nov 03 20:37:59.846292 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:37:59.846236    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg" podStartSLOduration=1.846221538 podStartE2EDuration="1.846221538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:59.844281073 +0000 UTC m=+2022.053297071" watchObservedRunningTime="2026-11-03 20:37:59.846221538 +0000 UTC m=+2022.055237526"
Nov 03 20:38:00.820863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:00.820827    5424 generic.go:356] "Generic (PLEG): container finished" podID="b0a6dee1-9af4-445d-b5b3-15f655ee0c57" containerID="8318e6ae2b346674a3f810aa79f133de7b04fe922cabd395b5bebd150369726f" exitCode=0
Nov 03 20:38:00.821273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:00.820895    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg" event={"ID":"b0a6dee1-9af4-445d-b5b3-15f655ee0c57","Type":"ContainerDied","Data":"8318e6ae2b346674a3f810aa79f133de7b04fe922cabd395b5bebd150369726f"}
Nov 03 20:38:01.965734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:01.965711    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:38:02.095609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.095505    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\") pod \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") "
Nov 03 20:38:02.095609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.095588    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\" (UniqueName: \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\") pod \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") "
Nov 03 20:38:02.095847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.095645    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\") pod \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\" (UID: \"b0a6dee1-9af4-445d-b5b3-15f655ee0c57\") "
Nov 03 20:38:02.096220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.096011    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca" pod "b0a6dee1-9af4-445d-b5b3-15f655ee0c57" (UID: "b0a6dee1-9af4-445d-b5b3-15f655ee0c57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:02.096220 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.096158    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:02.097755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.097720    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca" pod "b0a6dee1-9af4-445d-b5b3-15f655ee0c57" (UID: "b0a6dee1-9af4-445d-b5b3-15f655ee0c57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:02.097886 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.097865    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm" pod "b0a6dee1-9af4-445d-b5b3-15f655ee0c57" (UID: "b0a6dee1-9af4-445d-b5b3-15f655ee0c57"). InnerVolumeSpecName "kube-api-access-f9htm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:02.196567 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.196513    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:02.196700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.196593    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9htm\" (UniqueName: \"kubernetes.io/projected/b0a6dee1-9af4-445d-b5b3-15f655ee0c57-kube-api-access-f9htm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:02.834682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.834646    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg" event={"ID":"b0a6dee1-9af4-445d-b5b3-15f655ee0c57","Type":"ContainerDied","Data":"80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333"}
Nov 03 20:38:02.834682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.834669    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"
Nov 03 20:38:02.834902 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:02.834693    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bd44c74f6d1addbf16a0526a93be5edc50e37944367387d7958d3f17f39333"
Nov 03 20:38:03.009137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.009101    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:38:03.061707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.061658    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"]
Nov 03 20:38:03.062245 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.062222    5424 cpu_manager.go:414] "containerMap: removing container" podUID="b0a6dee1-9af4-445d-b5b3-15f655ee0c57" containerName="revoke-keys"
Nov 03 20:38:03.062331 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.062248    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a6dee1-9af4-445d-b5b3-15f655ee0c57" containerName="revoke-keys"
Nov 03 20:38:03.062387 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.062378    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a6dee1-9af4-445d-b5b3-15f655ee0c57" containerName="revoke-keys"
Nov 03 20:38:03.065826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.065804    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.081523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.081502    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:38:03.084653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.084631    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:38:03.087946 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.087893    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"]
Nov 03 20:38:03.104219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.104181    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4slq\" (UniqueName: \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.104363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.104342    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.104480 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.104465    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.121336 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.121306    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:38:03.184592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.184569    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:38:03.205060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.205034    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.205159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.205088    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4slq\" (UniqueName: \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.205159 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.205148    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.205710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.205683    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.207602 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.207586    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.227487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.227443    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4slq\" (UniqueName: \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\") pod \"maas-api-revoke-sub-3840d97c-xx2hg\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.382836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.382759    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:03.683035 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.683009    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"]
Nov 03 20:38:03.684799 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:38:03.684770    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565ffec3_4f8e_41e8_acd2_46ee0b724d54.slice/crio-bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d WatchSource:0}: Error finding container bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d: Status 404 returned error can't find the container with id bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d
Nov 03 20:38:03.842738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:03.842710    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg" event={"ID":"565ffec3-4f8e-41e8-acd2-46ee0b724d54","Type":"ContainerStarted","Data":"bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d"}
Nov 03 20:38:04.850362 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:04.850275    5424 generic.go:356] "Generic (PLEG): container finished" podID="565ffec3-4f8e-41e8-acd2-46ee0b724d54" containerID="17503f9d71dc094e7e4a228a9f2c233b683c9f2ec24efd9371741087feb13076" exitCode=0
Nov 03 20:38:04.850751 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:04.850358    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg" event={"ID":"565ffec3-4f8e-41e8-acd2-46ee0b724d54","Type":"ContainerDied","Data":"17503f9d71dc094e7e4a228a9f2c233b683c9f2ec24efd9371741087feb13076"}
Nov 03 20:38:05.994350 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:05.994327    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:06.037310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.037279    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\") pod \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") "
Nov 03 20:38:06.037457 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.037441    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\") pod \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") "
Nov 03 20:38:06.037604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.037581    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\" (UniqueName: \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\") pod \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\" (UID: \"565ffec3-4f8e-41e8-acd2-46ee0b724d54\") "
Nov 03 20:38:06.037998 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.037968    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca" pod "565ffec3-4f8e-41e8-acd2-46ee0b724d54" (UID: "565ffec3-4f8e-41e8-acd2-46ee0b724d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:06.038204 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.038187    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:06.039710 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.039680    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca" pod "565ffec3-4f8e-41e8-acd2-46ee0b724d54" (UID: "565ffec3-4f8e-41e8-acd2-46ee0b724d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:06.040187 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.040167    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq" pod "565ffec3-4f8e-41e8-acd2-46ee0b724d54" (UID: "565ffec3-4f8e-41e8-acd2-46ee0b724d54"). InnerVolumeSpecName "kube-api-access-k4slq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:06.138944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.138871    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565ffec3-4f8e-41e8-acd2-46ee0b724d54-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:06.138944 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.138898    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4slq\" (UniqueName: \"kubernetes.io/projected/565ffec3-4f8e-41e8-acd2-46ee0b724d54-kube-api-access-k4slq\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:06.872164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.872131    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"
Nov 03 20:38:06.872164 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.872148    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg" event={"ID":"565ffec3-4f8e-41e8-acd2-46ee0b724d54","Type":"ContainerDied","Data":"bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d"}
Nov 03 20:38:06.872397 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:06.872196    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcd3f40e7b48dbb6e7430f75afa713ce5d6bfe08ef443da7f13e8bb231e8c09d"
Nov 03 20:38:08.009774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.009716    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" podUID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" containerName="payload-pre-processing" containerID="cri-o://41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4" gracePeriod=25
Nov 03 20:38:08.276060 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.275977    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:38:08.357784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.357757    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\") pod \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") "
Nov 03 20:38:08.357784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.357790    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\" (UniqueName: \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\") pod \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\" (UID: \"7454d1f9-4e84-42d5-94d8-3b1dd9960841\") "
Nov 03 20:38:08.358217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.358192    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume" pod "7454d1f9-4e84-42d5-94d8-3b1dd9960841" (UID: "7454d1f9-4e84-42d5-94d8-3b1dd9960841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.360071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.360044    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h" pod "7454d1f9-4e84-42d5-94d8-3b1dd9960841" (UID: "7454d1f9-4e84-42d5-94d8-3b1dd9960841"). InnerVolumeSpecName "kube-api-access-kjh9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.458633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.458608    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7454d1f9-4e84-42d5-94d8-3b1dd9960841-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:08.458633 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.458633    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjh9h\" (UniqueName: \"kubernetes.io/projected/7454d1f9-4e84-42d5-94d8-3b1dd9960841-kube-api-access-kjh9h\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:08.873603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.873567    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:38:08.885584 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.885553    5424 generic.go:356] "Generic (PLEG): container finished" podID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" containerID="41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4" exitCode=0
Nov 03 20:38:08.885714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.885611    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"
Nov 03 20:38:08.885714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.885645    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" event={"ID":"7454d1f9-4e84-42d5-94d8-3b1dd9960841","Type":"ContainerDied","Data":"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"}
Nov 03 20:38:08.885714 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.885705    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8" event={"ID":"7454d1f9-4e84-42d5-94d8-3b1dd9960841","Type":"ContainerDied","Data":"f881e3df1b70423db629981be7c24de4534351e5f740417648606a0362900d00"}
Nov 03 20:38:08.885831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.885732    5424 scope.go:122] "RemoveContainer" containerID="41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"
Nov 03 20:38:08.895975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.895957    5424 scope.go:122] "RemoveContainer" containerID="41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"
Nov 03 20:38:08.896250 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:08.896232    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4\": container with ID starting with 41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4 not found: ID does not exist" containerID="41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"
Nov 03 20:38:08.896298 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.896259    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4"} err="failed to get container status \"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4\": rpc error: code = NotFound desc = could not find container \"41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4\": container with ID starting with 41cc812844de7139851684ffbdab57d2c0fe1f852ace15f738235da7050fe7d4 not found: ID does not exist"
Nov 03 20:38:08.952666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.952637    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:38:08.964846 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:08.964816    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e80203-6bd866d97f-b4tm8"]
Nov 03 20:38:09.917852 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:09.917816    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" path="/var/lib/kubelet/pods/7454d1f9-4e84-42d5-94d8-3b1dd9960841/volumes"
Nov 03 20:38:12.285818 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:12.285776    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:38:12.324953 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.324897    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" containerID="cri-o://18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325" gracePeriod=45
Nov 03 20:38:12.347671 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.347640    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main" containerID="cri-o://690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0" gracePeriod=45
Nov 03 20:38:12.362174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.362144    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Nov 03 20:38:12.591522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.591490    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:38:12.620754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.620732    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:38:12.700324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700293    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700349    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\" (UniqueName: \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700394    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.700495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700448    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700468    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700495 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700492    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700540    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700576    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700602    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\") pod \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\" (UID: \"cf2c2ad6-ab7b-471d-9311-7b1197f23afb\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700614    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700662    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700693    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\" (UniqueName: \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.700843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.700759    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\" (UniqueName: \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\") pod \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\" (UID: \"f47ace9d-2dfe-4a97-8a32-9e00d230ed44\") "
Nov 03 20:38:12.701378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701069    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.701378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701153    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.701378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701240    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.701378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701324    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.701378 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701327    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.701677 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.701655    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.703182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.703157    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "kube-api-access-fftzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:12.703371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.703348    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.703452 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.703362    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.703919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.703890    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs" pod "cf2c2ad6-ab7b-471d-9311-7b1197f23afb" (UID: "cf2c2ad6-ab7b-471d-9311-7b1197f23afb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:12.703919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.703905    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "kube-api-access-schlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:12.704625 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.704597    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs" pod "f47ace9d-2dfe-4a97-8a32-9e00d230ed44" (UID: "f47ace9d-2dfe-4a97-8a32-9e00d230ed44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802628    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftzm\" (UniqueName: \"kubernetes.io/projected/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-kube-api-access-fftzm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802659    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802670    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802678    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802687    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802695    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.802718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802724    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.803249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802732    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2c2ad6-ab7b-471d-9311-7b1197f23afb-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.803249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802740    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.803249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802747    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schlz\" (UniqueName: \"kubernetes.io/projected/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-kube-api-access-schlz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.803249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.802755    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f47ace9d-2dfe-4a97-8a32-9e00d230ed44-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.913004 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.912965    5424 generic.go:356] "Generic (PLEG): container finished" podID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerID="18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325" exitCode=0
Nov 03 20:38:12.913168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.913009    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"
Nov 03 20:38:12.913168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.913061    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" event={"ID":"f47ace9d-2dfe-4a97-8a32-9e00d230ed44","Type":"ContainerDied","Data":"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"}
Nov 03 20:38:12.913168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.913104    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf" event={"ID":"f47ace9d-2dfe-4a97-8a32-9e00d230ed44","Type":"ContainerDied","Data":"8557ce5633b18907ec359947f0619e5f4c84ecb9d15e9726bd260bb5e7aee640"}
Nov 03 20:38:12.913168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.913125    5424 scope.go:122] "RemoveContainer" containerID="18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"
Nov 03 20:38:12.914682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.914644    5424 generic.go:356] "Generic (PLEG): container finished" podID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerID="690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0" exitCode=0
Nov 03 20:38:12.914806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.914694    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"
Nov 03 20:38:12.914806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.914720    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" event={"ID":"cf2c2ad6-ab7b-471d-9311-7b1197f23afb","Type":"ContainerDied","Data":"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"}
Nov 03 20:38:12.914806 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.914747    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk" event={"ID":"cf2c2ad6-ab7b-471d-9311-7b1197f23afb","Type":"ContainerDied","Data":"40cc6fc3cf5d1f9c5c64d5ba7709fdc0ba28ff7c884288a60e8d98544c8733f2"}
Nov 03 20:38:12.923977 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.923959    5424 scope.go:122] "RemoveContainer" containerID="18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"
Nov 03 20:38:12.924438 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:12.924283    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325\": container with ID starting with 18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325 not found: ID does not exist" containerID="18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"
Nov 03 20:38:12.924438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.924317    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325"} err="failed to get container status \"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325\": rpc error: code = NotFound desc = could not find container \"18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325\": container with ID starting with 18df98ef9bea1870ccb4d33a25ed2f792d6bab4000e170833c9614ca233f9325 not found: ID does not exist"
Nov 03 20:38:12.924438 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.924336    5424 scope.go:122] "RemoveContainer" containerID="690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"
Nov 03 20:38:12.933470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.933450    5424 scope.go:122] "RemoveContainer" containerID="690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"
Nov 03 20:38:12.933760 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:12.933741    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0\": container with ID starting with 690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0 not found: ID does not exist" containerID="690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"
Nov 03 20:38:12.933807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.933767    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0"} err="failed to get container status \"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0\": rpc error: code = NotFound desc = could not find container \"690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0\": container with ID starting with 690064262ef734b2b330c2ba35a7ae2e8dc7b25da6691d8ece8b6314671c9de0 not found: ID does not exist"
Nov 03 20:38:12.966182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.966157    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:38:12.978332 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:12.978306    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-w4-e80203-kserve-5cb7877787-8tfjf"]
Nov 03 20:38:13.012388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.012363    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:38:13.014488 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.014469    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/simulator-w4-e80203-kserve-5bd485cb8f-8lkhk"]
Nov 03 20:38:13.823503 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.823465    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"]
Nov 03 20:38:13.835470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.835441    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d1bfb8-7l2zg"]
Nov 03 20:38:13.874600 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.874382    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" podUID="e7e42741-62a4-4a34-a9a2-086f76971297" containerName="payload-processing" containerID="cri-o://d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c" gracePeriod=25
Nov 03 20:38:13.918140 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.918109    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a6dee1-9af4-445d-b5b3-15f655ee0c57" path="/var/lib/kubelet/pods/b0a6dee1-9af4-445d-b5b3-15f655ee0c57/volumes"
Nov 03 20:38:13.918504 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.918491    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" path="/var/lib/kubelet/pods/cf2c2ad6-ab7b-471d-9311-7b1197f23afb/volumes"
Nov 03 20:38:13.918995 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:13.918977    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" path="/var/lib/kubelet/pods/f47ace9d-2dfe-4a97-8a32-9e00d230ed44/volumes"
Nov 03 20:38:14.137157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.137134    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:38:14.215178 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.215145    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\" (UniqueName: \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\") pod \"e7e42741-62a4-4a34-a9a2-086f76971297\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") "
Nov 03 20:38:14.215363 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.215215    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\") pod \"e7e42741-62a4-4a34-a9a2-086f76971297\" (UID: \"e7e42741-62a4-4a34-a9a2-086f76971297\") "
Nov 03 20:38:14.215563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.215522    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume" pod "e7e42741-62a4-4a34-a9a2-086f76971297" (UID: "e7e42741-62a4-4a34-a9a2-086f76971297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:14.215668 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.215650    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e42741-62a4-4a34-a9a2-086f76971297-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:14.217442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.217413    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5" pod "e7e42741-62a4-4a34-a9a2-086f76971297" (UID: "e7e42741-62a4-4a34-a9a2-086f76971297"). InnerVolumeSpecName "kube-api-access-tvcd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:14.316157 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.316124    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcd5\" (UniqueName: \"kubernetes.io/projected/e7e42741-62a4-4a34-a9a2-086f76971297-kube-api-access-tvcd5\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:14.931249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.931210    5424 generic.go:356] "Generic (PLEG): container finished" podID="e7e42741-62a4-4a34-a9a2-086f76971297" containerID="d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c" exitCode=0
Nov 03 20:38:14.931769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.931257    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"
Nov 03 20:38:14.931769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.931302    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" event={"ID":"e7e42741-62a4-4a34-a9a2-086f76971297","Type":"ContainerDied","Data":"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"}
Nov 03 20:38:14.931769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.931361    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7" event={"ID":"e7e42741-62a4-4a34-a9a2-086f76971297","Type":"ContainerDied","Data":"750610490b46c450bd04bd4f4ebfcc59cf6ed1ad87a6903150278bf2c15e0beb"}
Nov 03 20:38:14.931769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.931390    5424 scope.go:122] "RemoveContainer" containerID="d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"
Nov 03 20:38:14.941607 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.941588    5424 scope.go:122] "RemoveContainer" containerID="d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"
Nov 03 20:38:14.941885 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:14.941862    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c\": container with ID starting with d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c not found: ID does not exist" containerID="d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"
Nov 03 20:38:14.941961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.941894    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c"} err="failed to get container status \"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c\": rpc error: code = NotFound desc = could not find container \"d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c\": container with ID starting with d815bb1b4c84db5d06f93c140ca13bfc666cd5f1e97ae867ec719268c2d52b6c not found: ID does not exist"
Nov 03 20:38:14.982541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.982505    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:38:14.994463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:14.994442    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d1bfb8-575f56d998-hzrh7"]
Nov 03 20:38:15.917804 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:15.917769    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e42741-62a4-4a34-a9a2-086f76971297" path="/var/lib/kubelet/pods/e7e42741-62a4-4a34-a9a2-086f76971297/volumes"
Nov 03 20:38:15.956322 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:15.956284    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"]
Nov 03 20:38:15.965001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:15.964968    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e80203-s7g2h"]
Nov 03 20:38:17.919523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:17.919492    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7101469a-e5bd-4e70-92d5-ec752d8192ed" path="/var/lib/kubelet/pods/7101469a-e5bd-4e70-92d5-ec752d8192ed/volumes"
Nov 03 20:38:21.612132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.612101    5424 scope.go:122] "RemoveContainer" containerID="128849096f47092b3ba881bacabc9d3b85cb6e5a9ac801e570287caaa9190932"
Nov 03 20:38:21.621843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.621818    5424 scope.go:122] "RemoveContainer" containerID="7a4de68ca1fa3e766bfad53a5759c441e14b8c12a169be20f2c6df244981619f"
Nov 03 20:38:21.630770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.630745    5424 scope.go:122] "RemoveContainer" containerID="f5b1a7579ce7498b6b110a09bd77535a09c567b81d629770fff846bc40d9424a"
Nov 03 20:38:21.640745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.640722    5424 scope.go:122] "RemoveContainer" containerID="f6d4ba09fc01311b7176711186fa6c4f7d1db5f6a68be6bbd7c62d51b51c9918"
Nov 03 20:38:21.649370 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.649340    5424 scope.go:122] "RemoveContainer" containerID="30e4a7d3713c07ad69a3513c446d6eea486c2f622058edf1222a71d96c8d908c"
Nov 03 20:38:21.657971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:21.657947    5424 scope.go:122] "RemoveContainer" containerID="28b08d1ece9bd011dbe2b2aefdd5c5c609948afc1e95b7aee46a118a03a6636b"
Nov 03 20:38:23.097522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:23.097465    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:38:23.098992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:23.098933    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy" containerID="cri-o://b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b" gracePeriod=30
Nov 03 20:38:26.222904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:26.222873    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:38:28.367487 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.367462    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:38:28.455778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455694    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.455778 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455738    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455845    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455879    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455902    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456018 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.455929    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456170    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.456280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456235    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:28.456352 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456318    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.456485 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456461    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456826 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456352    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.456905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456880    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.456971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.456954    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\") pod \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\" (UID: \"66efb845-9ea2-422b-9be1-42ddfabeaf7b\") "
Nov 03 20:38:28.458193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.457567    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.458809    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.458848    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.458864    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.458880    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.458897    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.462933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.460192    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.470316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.470127    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:28.470408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.470343    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:28.470884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.470861    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8" pod "66efb845-9ea2-422b-9be1-42ddfabeaf7b" (UID: "66efb845-9ea2-422b-9be1-42ddfabeaf7b"). InnerVolumeSpecName "kube-api-access-9fsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:28.559524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.559488    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.559524 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.559519    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsx8\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-kube-api-access-9fsx8\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.559711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.559562    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.559711 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:28.559573    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efb845-9ea2-422b-9be1-42ddfabeaf7b-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:29.027577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.027542    5424 generic.go:356] "Generic (PLEG): container finished" podID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerID="b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b" exitCode=0
Nov 03 20:38:29.027762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.027625    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"
Nov 03 20:38:29.027762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.027628    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" event={"ID":"66efb845-9ea2-422b-9be1-42ddfabeaf7b","Type":"ContainerDied","Data":"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"}
Nov 03 20:38:29.027762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.027692    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm" event={"ID":"66efb845-9ea2-422b-9be1-42ddfabeaf7b","Type":"ContainerDied","Data":"f5f7a6f7d2d75ceaf26438d272a9283abc3ee8f8e5be0d64aea733ba465c0814"}
Nov 03 20:38:29.027762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.027723    5424 scope.go:122] "RemoveContainer" containerID="b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"
Nov 03 20:38:29.037973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.037955    5424 scope.go:122] "RemoveContainer" containerID="b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"
Nov 03 20:38:29.038258 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:29.038238    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b\": container with ID starting with b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b not found: ID does not exist" containerID="b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"
Nov 03 20:38:29.038335 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.038267    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b"} err="failed to get container status \"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b\": rpc error: code = NotFound desc = could not find container \"b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b\": container with ID starting with b63b676dbdc517a1bfe2097b242240fdab4c288d0264a0de6ab11491119a420b not found: ID does not exist"
Nov 03 20:38:29.079377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.079353    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:38:29.091683 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.091663    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-4wbcm"]
Nov 03 20:38:29.917470 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:29.917430    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" path="/var/lib/kubelet/pods/66efb845-9ea2-422b-9be1-42ddfabeaf7b/volumes"
Nov 03 20:38:33.432436 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432390    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"]
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432790    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432803    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432813    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e7e42741-62a4-4a34-a9a2-086f76971297" containerName="payload-processing"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432820    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e42741-62a4-4a34-a9a2-086f76971297" containerName="payload-processing"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432831    5424 cpu_manager.go:414] "containerMap: removing container" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432836    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432846    5424 cpu_manager.go:414] "containerMap: removing container" podUID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" containerName="payload-pre-processing"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432851    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" containerName="payload-pre-processing"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432867    5424 cpu_manager.go:414] "containerMap: removing container" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432872    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432879    5424 cpu_manager.go:414] "containerMap: removing container" podUID="565ffec3-4f8e-41e8-acd2-46ee0b724d54" containerName="revoke-keys"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432883    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565ffec3-4f8e-41e8-acd2-46ee0b724d54" containerName="revoke-keys"
Nov 03 20:38:33.432926 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432944    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47ace9d-2dfe-4a97-8a32-9e00d230ed44" containerName="main"
Nov 03 20:38:33.433680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432953    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e42741-62a4-4a34-a9a2-086f76971297" containerName="payload-processing"
Nov 03 20:38:33.433680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432962    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="66efb845-9ea2-422b-9be1-42ddfabeaf7b" containerName="istio-proxy"
Nov 03 20:38:33.433680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432973    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="7454d1f9-4e84-42d5-94d8-3b1dd9960841" containerName="payload-pre-processing"
Nov 03 20:38:33.433680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432981    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2c2ad6-ab7b-471d-9311-7b1197f23afb" containerName="main"
Nov 03 20:38:33.433680 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.432986    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="565ffec3-4f8e-41e8-acd2-46ee0b724d54" containerName="revoke-keys"
Nov 03 20:38:33.437510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.437489    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.456338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.456310    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"]
Nov 03 20:38:33.504970 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.504930    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhqq\" (UniqueName: \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.505150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.505099    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.505150 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.505134    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.536659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.536625    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:38:33.536659 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.536625    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:38:33.536853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.536664    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:38:33.536853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.536650    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:38:33.606847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.606810    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.607056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.606864    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.607056 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.606910    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhqq\" (UniqueName: \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.607766 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.607736    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.609942 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.609912    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.628729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.628689    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhqq\" (UniqueName: \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\") pod \"maas-api-revoke-sub-f9905f40-gs5tl\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:33.751769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:33.751734    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:34.036675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:34.036570    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"]
Nov 03 20:38:34.038964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:34.038930    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"]
Nov 03 20:38:34.048146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:34.048117    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fea5d1-w72mg"]
Nov 03 20:38:34.050224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:34.050194    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659534aa-xnqhs"]
Nov 03 20:38:34.059991 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:34.059958    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"]
Nov 03 20:38:34.065486 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:38:34.065461    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf005f8_e910_415c_a939_04414234e9cf.slice/crio-1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67 WatchSource:0}: Error finding container 1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67: Status 404 returned error can't find the container with id 1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67
Nov 03 20:38:35.063185 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:35.063151    5424 generic.go:356] "Generic (PLEG): container finished" podID="9cf005f8-e910-415c-a939-04414234e9cf" containerID="c8990264cece55116888110a6c9b293ffd5b3d8f94dfefcfd93332b5d59d27da" exitCode=0
Nov 03 20:38:35.063638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:35.063232    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl" event={"ID":"9cf005f8-e910-415c-a939-04414234e9cf","Type":"ContainerDied","Data":"c8990264cece55116888110a6c9b293ffd5b3d8f94dfefcfd93332b5d59d27da"}
Nov 03 20:38:35.063638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:35.063306    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl" event={"ID":"9cf005f8-e910-415c-a939-04414234e9cf","Type":"ContainerStarted","Data":"1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67"}
Nov 03 20:38:35.917964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:35.917925    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e" path="/var/lib/kubelet/pods/4cdcd131-c5f8-4d9b-aaa1-7e35182fb01e/volumes"
Nov 03 20:38:35.918343 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:35.918327    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b219ae6c-acf8-4bde-a986-fef60824fada" path="/var/lib/kubelet/pods/b219ae6c-acf8-4bde-a986-fef60824fada/volumes"
Nov 03 20:38:36.215724 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.215693    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:36.331838 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.331800    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\" (UniqueName: \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\") pod \"9cf005f8-e910-415c-a939-04414234e9cf\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") "
Nov 03 20:38:36.332051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.331844    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\") pod \"9cf005f8-e910-415c-a939-04414234e9cf\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") "
Nov 03 20:38:36.332051 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.331888    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\") pod \"9cf005f8-e910-415c-a939-04414234e9cf\" (UID: \"9cf005f8-e910-415c-a939-04414234e9cf\") "
Nov 03 20:38:36.332305 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.332277    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca" pod "9cf005f8-e910-415c-a939-04414234e9cf" (UID: "9cf005f8-e910-415c-a939-04414234e9cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:36.334194 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.334156    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca" pod "9cf005f8-e910-415c-a939-04414234e9cf" (UID: "9cf005f8-e910-415c-a939-04414234e9cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:36.334290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.334198    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq" pod "9cf005f8-e910-415c-a939-04414234e9cf" (UID: "9cf005f8-e910-415c-a939-04414234e9cf"). InnerVolumeSpecName "kube-api-access-2qhqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:36.433415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.433375    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhqq\" (UniqueName: \"kubernetes.io/projected/9cf005f8-e910-415c-a939-04414234e9cf-kube-api-access-2qhqq\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:36.433415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.433408    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:36.433415 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:36.433420    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cf005f8-e910-415c-a939-04414234e9cf-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:37.076353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:37.076317    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"
Nov 03 20:38:37.076556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:37.076322    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl" event={"ID":"9cf005f8-e910-415c-a939-04414234e9cf","Type":"ContainerDied","Data":"1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67"}
Nov 03 20:38:37.076556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:37.076467    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2d9589f033ef6ccfb49d9cc253ccf01ad0039b54e9fbcfea4d3101e1471a67"
Nov 03 20:38:41.243627 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.243588    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main" containerID="cri-o://11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355" gracePeriod=45
Nov 03 20:38:41.492656 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.492633    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:38:41.580795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.580720    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.580795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.580766    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.581043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.580844    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.581043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.580872    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.581043 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.580994    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\" (UniqueName: \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.581214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581065    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\") pod \"6c76d46c-e0b6-466a-9de2-28414bac6a54\" (UID: \"6c76d46c-e0b6-466a-9de2-28414bac6a54\") "
Nov 03 20:38:41.581214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581052    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.581214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581127    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.581214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581136    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.581465 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581446    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.581544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581470    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.581544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.581484    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.583502 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.583472    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:41.583665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.583498    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "kube-api-access-pxm7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:41.583741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.583716    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm" pod "6c76d46c-e0b6-466a-9de2-28414bac6a54" (UID: "6c76d46c-e0b6-466a-9de2-28414bac6a54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.690285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.683192    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76d46c-e0b6-466a-9de2-28414bac6a54-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.690285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.683279    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76d46c-e0b6-466a-9de2-28414bac6a54-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.690285 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:41.683308    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxm7k\" (UniqueName: \"kubernetes.io/projected/6c76d46c-e0b6-466a-9de2-28414bac6a54-kube-api-access-pxm7k\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:38:42.105143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.105112    5424 generic.go:356] "Generic (PLEG): container finished" podID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerID="11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355" exitCode=0
Nov 03 20:38:42.105328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.105178    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"
Nov 03 20:38:42.105328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.105199    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" event={"ID":"6c76d46c-e0b6-466a-9de2-28414bac6a54","Type":"ContainerDied","Data":"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"}
Nov 03 20:38:42.105328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.105244    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns" event={"ID":"6c76d46c-e0b6-466a-9de2-28414bac6a54","Type":"ContainerDied","Data":"18af113f28af5d6094ad1e0f64c2e53ff805205cb6f7c604bb34ddfdb7d8e8e2"}
Nov 03 20:38:42.105328 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.105266    5424 scope.go:122] "RemoveContainer" containerID="11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"
Nov 03 20:38:42.114493 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.114474    5424 scope.go:122] "RemoveContainer" containerID="11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"
Nov 03 20:38:42.114821 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:42.114800    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355\": container with ID starting with 11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355 not found: ID does not exist" containerID="11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"
Nov 03 20:38:42.114894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.114832    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355"} err="failed to get container status \"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355\": rpc error: code = NotFound desc = could not find container \"11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355\": container with ID starting with 11830c8ad4465d5674ece640ccd4e22510392370c5bbb94890236bdac7165355 not found: ID does not exist"
Nov 03 20:38:42.156353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.156320    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:38:42.168867 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:42.168832    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-482722/simulator-w3-482722-kserve-57cdd8f99f-ldrns"]
Nov 03 20:38:42.285941 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:38:42.285906    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:38:43.917779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:43.917748    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" path="/var/lib/kubelet/pods/6c76d46c-e0b6-466a-9de2-28414bac6a54/volumes"
Nov 03 20:38:45.027448 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.027406    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"]
Nov 03 20:38:45.040420 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.040372    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"]
Nov 03 20:38:45.042383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.042357    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce99c8d-rbz76"]
Nov 03 20:38:45.044017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.043994    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fab1bb26-4t9h7"]
Nov 03 20:38:45.918908 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.918868    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267a1961-2acb-435e-b8cb-e2937570c03c" path="/var/lib/kubelet/pods/267a1961-2acb-435e-b8cb-e2937570c03c/volumes"
Nov 03 20:38:45.919446 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:45.919427    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772286d7-8bcd-4fb6-a143-ade7c5d4488c" path="/var/lib/kubelet/pods/772286d7-8bcd-4fb6-a143-ade7c5d4488c/volumes"
Nov 03 20:38:50.027510 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:50.027475    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"]
Nov 03 20:38:50.041040 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:50.040999    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e194f11-lj5n6"]
Nov 03 20:38:51.026629 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:51.026595    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"]
Nov 03 20:38:51.038849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:51.038822    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-008f4600-5mbx7"]
Nov 03 20:38:51.918252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:51.918220    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f4f692-d011-4003-b79e-e16f1b9d2d4d" path="/var/lib/kubelet/pods/03f4f692-d011-4003-b79e-e16f1b9d2d4d/volumes"
Nov 03 20:38:51.918640 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:51.918627    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c27e32-3acd-410b-bcfe-f82eb4dd24e0" path="/var/lib/kubelet/pods/44c27e32-3acd-410b-bcfe-f82eb4dd24e0/volumes"
Nov 03 20:38:55.597201 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:55.597164    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:38:55.600918 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:38:55.600853    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy" containerID="cri-o://dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44" gracePeriod=30
Nov 03 20:39:00.897280 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.897246    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:39:00.960664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960625    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.960873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960758    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.960873 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960860    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960901    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960937    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.960974    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961029 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.961010    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.961038    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.961273 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.961066    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\") pod \"a0ca4640-8625-4d65-97cd-5742801f27c0\" (UID: \"a0ca4640-8625-4d65-97cd-5742801f27c0\") "
Nov 03 20:39:00.962381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.962096    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.962381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.962121    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.962381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.962202    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.962795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.962758    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.963069 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.963040    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:00.965626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.965403    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.965626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.965500    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:00.965626 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.965598    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "kube-api-access-mnzls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:00.966028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:00.965999    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo" pod "a0ca4640-8625-4d65-97cd-5742801f27c0" (UID: "a0ca4640-8625-4d65-97cd-5742801f27c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:01.062573 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062502    5424 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-envoy\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062577    5424 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-podinfo\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062597    5424 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062611    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062624    5424 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca4640-8625-4d65-97cd-5742801f27c0-istiod-ca-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062637    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzls\" (UniqueName: \"kubernetes.io/projected/a0ca4640-8625-4d65-97cd-5742801f27c0-kube-api-access-mnzls\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062654    5424 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-workload-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062668    5424 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-credential-socket\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.062762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.062684    5424 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca4640-8625-4d65-97cd-5742801f27c0-istio-data\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.221577 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.221466    5424 generic.go:356] "Generic (PLEG): container finished" podID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerID="dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44" exitCode=0
Nov 03 20:39:01.221761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.221571    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"
Nov 03 20:39:01.221761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.221568    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerDied","Data":"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"}
Nov 03 20:39:01.221761 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.221735    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg" event={"ID":"a0ca4640-8625-4d65-97cd-5742801f27c0","Type":"ContainerDied","Data":"145ffc3025412c165f2247a95a93c6a696e95a50d4de7afea09cc4418eb87fc2"}
Nov 03 20:39:01.221907 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.221772    5424 scope.go:122] "RemoveContainer" containerID="dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"
Nov 03 20:39:01.232860 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.232818    5424 scope.go:122] "RemoveContainer" containerID="cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"
Nov 03 20:39:01.242777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.242751    5424 scope.go:122] "RemoveContainer" containerID="dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"
Nov 03 20:39:01.243137 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:39:01.243117    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44\": container with ID starting with dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44 not found: ID does not exist" containerID="dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"
Nov 03 20:39:01.243223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.243174    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44"} err="failed to get container status \"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44\": rpc error: code = NotFound desc = could not find container \"dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44\": container with ID starting with dabf7ec99986b5612adfc68565a8d3dce06081f674ea28c532366f73dc06ea44 not found: ID does not exist"
Nov 03 20:39:01.243223 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.243202    5424 scope.go:122] "RemoveContainer" containerID="cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"
Nov 03 20:39:01.243498 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:39:01.243481    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97\": container with ID starting with cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97 not found: ID does not exist" containerID="cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"
Nov 03 20:39:01.243609 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.243506    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97"} err="failed to get container status \"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97\": rpc error: code = NotFound desc = could not find container \"cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97\": container with ID starting with cbb78faf68e3284a78d48deb7da0f542d1f9d65ff22d063cede2b5bc711bbd97 not found: ID does not exist"
Nov 03 20:39:01.275193 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.275158    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:39:01.287453 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.287423    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-6dcdg"]
Nov 03 20:39:01.922561 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:01.918997    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" path="/var/lib/kubelet/pods/a0ca4640-8625-4d65-97cd-5742801f27c0/volumes"
Nov 03 20:39:10.032734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:10.032692    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"]
Nov 03 20:39:10.045186 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:10.045154    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b51689-p8bt8"]
Nov 03 20:39:11.918071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:11.918033    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6216744-8e3b-4f88-b670-e5ee474624e0" path="/var/lib/kubelet/pods/b6216744-8e3b-4f88-b670-e5ee474624e0/volumes"
Nov 03 20:39:12.285033 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:39:12.284997    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:39:21.749182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:21.749149    5424 scope.go:122] "RemoveContainer" containerID="ac7fe97fee657b7a05d9cbe111221e782ab6e41c1b8d420549eab6b25e50bd75"
Nov 03 20:39:21.758439 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:21.758416    5424 scope.go:122] "RemoveContainer" containerID="e04e4c102df5c9ee6e0644d24876c881f349ed52bf0d8f6a3ae2923ba541549d"
Nov 03 20:39:21.766715 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:21.766696    5424 scope.go:122] "RemoveContainer" containerID="d6b98b2f9b11113721666527c371a98e7ff1583f3caf4d52a859078ee48ed33f"
Nov 03 20:39:21.774774 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:21.774756    5424 scope.go:122] "RemoveContainer" containerID="340d3541a1e6be6a2985b17ff377e5df128ecb2d1366b8c1b8668fb4387ba6c5"
Nov 03 20:39:21.785210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:21.785185    5424 scope.go:122] "RemoveContainer" containerID="7472554314a2e95a23997b8ca67f36a315f67c668bd4f5c0ba63690aff918642"
Nov 03 20:39:32.318700 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.318659    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:39:32.319154 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.318912    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerName="discovery" containerID="cri-o://a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf" gracePeriod=30
Nov 03 20:39:32.571111 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.571048    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:39:32.680755 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.680718    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.680964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.680772    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.680964 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.680798    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.681095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.680983    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.681095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.681028    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.681095 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.681080    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.681272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.681212    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\") pod \"e4ef0547-476a-4b95-a7ea-8d38681afc88\" (UID: \"e4ef0547-476a-4b95-a7ea-8d38681afc88\") "
Nov 03 20:39:32.681646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.681478    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:32.681646 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.681602    5424 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-ca-configmap\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.683445 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.683415    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "kube-api-access-kg8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:32.683592 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.683409    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:32.683765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.683737    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:32.684023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.683995    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:32.684023 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.684013    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:32.684149 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.683994    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs" pod "e4ef0547-476a-4b95-a7ea-8d38681afc88" (UID: "e4ef0547-476a-4b95-a7ea-8d38681afc88"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:32.782808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782767    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8kv\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-kube-api-access-kg8kv\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.782808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782800    5424 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-token\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.782808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782811    5424 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-cacerts\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.782808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782819    5424 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-kubeconfig\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.783100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782827    5424 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ef0547-476a-4b95-a7ea-8d38681afc88-local-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:32.783100 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:32.782836    5424 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4ef0547-476a-4b95-a7ea-8d38681afc88-istio-csr-dns-cert\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:39:33.429709 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.429665    5424 generic.go:356] "Generic (PLEG): container finished" podID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerID="a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf" exitCode=0
Nov 03 20:39:33.430145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.429718    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"
Nov 03 20:39:33.430145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.429746    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" event={"ID":"e4ef0547-476a-4b95-a7ea-8d38681afc88","Type":"ContainerDied","Data":"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"}
Nov 03 20:39:33.430145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.429798    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7" event={"ID":"e4ef0547-476a-4b95-a7ea-8d38681afc88","Type":"ContainerDied","Data":"519664b9d7ba8d34b26148ecd4df394b16257884a1331f67d84eaa4153b58111"}
Nov 03 20:39:33.430145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.429819    5424 scope.go:122] "RemoveContainer" containerID="a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"
Nov 03 20:39:33.440814 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.440793    5424 scope.go:122] "RemoveContainer" containerID="a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"
Nov 03 20:39:33.441112 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:39:33.441092    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf\": container with ID starting with a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf not found: ID does not exist" containerID="a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"
Nov 03 20:39:33.441172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.441122    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf"} err="failed to get container status \"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf\": rpc error: code = NotFound desc = could not find container \"a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf\": container with ID starting with a4b2aed4e8dd82cba5aa5fe6dfd84838778eaaa294329087721f0266ba1d4dcf not found: ID does not exist"
Nov 03 20:39:33.485104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.485073    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:39:33.490046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.490017    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r76c7"]
Nov 03 20:39:33.917507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:39:33.917472    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" path="/var/lib/kubelet/pods/e4ef0547-476a-4b95-a7ea-8d38681afc88/volumes"
Nov 03 20:39:42.285303 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:39:42.285268    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:40:12.285032 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:40:12.284990    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:40:21.834960 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.834932    5424 scope.go:122] "RemoveContainer" containerID="b585238f5aa02422a0c69efafd67d65476fd0818b35d34d8bd70e214431eed9f"
Nov 03 20:40:21.844121 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.844098    5424 scope.go:122] "RemoveContainer" containerID="53de101d0614481b5594fca464559ea18d89567bdc061510e63b52fd37b8f384"
Nov 03 20:40:21.852191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.852174    5424 scope.go:122] "RemoveContainer" containerID="0d852df7ec6571574b2ecd32fa8c853a39546f2a79b6773f6ada7f5797e5eb3e"
Nov 03 20:40:21.859954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.859927    5424 scope.go:122] "RemoveContainer" containerID="967829728ad8fcd1c537216785a76375ed557cfebfa877ba6f23d3bd6671987a"
Nov 03 20:40:21.869666 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.869642    5424 scope.go:122] "RemoveContainer" containerID="a4d9f8aba785fc8113a76423e2cc00a1351844b6639a9ed31864967a29d0e8b8"
Nov 03 20:40:21.877594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.877575    5424 scope.go:122] "RemoveContainer" containerID="fc4250b276e82648fe82bc00ceec0f16b7ba5238de5466d1965070905eb80aa7"
Nov 03 20:40:21.885119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.885101    5424 scope.go:122] "RemoveContainer" containerID="d325f76478f072d975f241ef29272d311bfea8a1a755b4d9fcf85038187fb49b"
Nov 03 20:40:21.892402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.892385    5424 scope.go:122] "RemoveContainer" containerID="503b2da7bd96f94f46a5e39d50f02226807969d99f521244d6073877ae432b08"
Nov 03 20:40:21.899777 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:40:21.899760    5424 scope.go:122] "RemoveContainer" containerID="32589e7d03729e7848b760c28b2e4f370970e4c75e315d50c4d5c73c8f668698"
Nov 03 20:40:42.283999 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:40:42.283921    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:41:05.026165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:05.026129    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"]
Nov 03 20:41:05.038853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:05.038826    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd00d862-xwqbs"]
Nov 03 20:41:05.917104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:05.917073    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c038e5be-340a-4896-a265-65c9552cdfe9" path="/var/lib/kubelet/pods/c038e5be-340a-4896-a265-65c9552cdfe9/volumes"
Nov 03 20:41:11.891738 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:11.891693    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:41:11.892173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:11.891784    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:41:12.285744 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:41:12.285707    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:41:17.027660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:17.027595    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"]
Nov 03 20:41:17.040066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:17.040041    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16eb999f-qj8b8"]
Nov 03 20:41:17.918016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:17.917985    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb09928-62ac-403b-8683-7d01c47467a0" path="/var/lib/kubelet/pods/fbb09928-62ac-403b-8683-7d01c47467a0/volumes"
Nov 03 20:41:27.025388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:27.025354    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"]
Nov 03 20:41:27.038118 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:27.038091    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00960573-lm8b9"]
Nov 03 20:41:27.917802 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:41:27.917760    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537d76b8-9231-49d9-be89-4a9670bfe08e" path="/var/lib/kubelet/pods/537d76b8-9231-49d9-be89-4a9670bfe08e/volumes"
Nov 03 20:41:42.285898 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:41:42.285859    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:42:12.284509 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:42:12.284411    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:42:21.957708 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:42:21.957670    5424 scope.go:122] "RemoveContainer" containerID="5adb45d4a76034995fd8a4f65ace5b1717f336652de10dad2c5a19267174621e"
Nov 03 20:42:21.966484 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:42:21.966460    5424 scope.go:122] "RemoveContainer" containerID="c4e16b7ffc0817180deb37be5d20e913704fb737b69df43a4d15a7bf4fdaf91f"
Nov 03 20:42:42.285655 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:42:42.285620    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:43:07.031741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:07.031709    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"]
Nov 03 20:43:07.044315 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:07.044286    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3840d97c-xx2hg"]
Nov 03 20:43:07.917608 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:07.917574    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565ffec3-4f8e-41e8-acd2-46ee0b724d54" path="/var/lib/kubelet/pods/565ffec3-4f8e-41e8-acd2-46ee0b724d54/volumes"
Nov 03 20:43:12.285110 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:43:12.285062    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:43:21.824698 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:21.824657    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:43:21.825253 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:21.824719    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:43:21.989456 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:21.989425    5424 scope.go:122] "RemoveContainer" containerID="6950a9f9ae2fcfd1eb57d367e799e2460ecd2a90cb9190886f773efdd71b9c18"
Nov 03 20:43:21.997904 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:21.997881    5424 scope.go:122] "RemoveContainer" containerID="7bd46e7bd61c152e17a85e8cdc971b4a7f3117e056f20bbc8020ec2a8f304d0b"
Nov 03 20:43:37.027102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:37.027026    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"]
Nov 03 20:43:37.040217 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:37.040187    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9905f40-gs5tl"]
Nov 03 20:43:37.917377 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:37.917343    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf005f8-e910-415c-a939-04414234e9cf" path="/var/lib/kubelet/pods/9cf005f8-e910-415c-a939-04414234e9cf/volumes"
Nov 03 20:43:42.285691 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:43:42.285658    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:43:56.895734 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:56.895688    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:43:56.896507 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:43:56.896474    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:44:12.285214 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:44:12.285174    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:44:22.019078 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:44:22.019042    5424 scope.go:122] "RemoveContainer" containerID="8318e6ae2b346674a3f810aa79f133de7b04fe922cabd395b5bebd150369726f"
Nov 03 20:44:22.027523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:44:22.027502    5424 scope.go:122] "RemoveContainer" containerID="17503f9d71dc094e7e4a228a9f2c233b683c9f2ec24efd9371741087feb13076"
Nov 03 20:44:22.035296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:44:22.035277    5424 scope.go:122] "RemoveContainer" containerID="429d1ae092cb46921b5a52bf90682f4adecba116f54692644c3674bcc2da6e65"
Nov 03 20:44:42.284767 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:44:42.284729    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:45:00.144404 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144327    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144663    5424 cpu_manager.go:414] "containerMap: removing container" podUID="9cf005f8-e910-415c-a939-04414234e9cf" containerName="revoke-keys"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144674    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf005f8-e910-415c-a939-04414234e9cf" containerName="revoke-keys"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144683    5424 cpu_manager.go:414] "containerMap: removing container" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerName="discovery"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144688    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerName="discovery"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144704    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144709    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144721    5424 cpu_manager.go:414] "containerMap: removing container" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144726    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144781    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf005f8-e910-415c-a939-04414234e9cf" containerName="revoke-keys"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144790    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ef0547-476a-4b95-a7ea-8d38681afc88" containerName="discovery"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144797    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:00.144849 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.144804    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c76d46c-e0b6-466a-9de2-28414bac6a54" containerName="main"
Nov 03 20:45:00.147541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.147515    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.162763 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.162742    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:45:00.164131 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.164114    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:00.175811 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.175787    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:00.178127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.178104    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:00.277682 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.277653    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xfg\" (UniqueName: \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.277832 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.277707    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.379013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.378982    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xfg\" (UniqueName: \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.379133 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.379038    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.379686 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.379667    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.399432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.399373    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xfg\" (UniqueName: \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\") pod \"maas-api-key-cleanup-29895645-kg9l6\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.459289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.459269    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:00.756685 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:00.756665    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:00.759272 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:45:00.759246    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d96139_652d_46cc_a98d_ac3d9301b890.slice/crio-acea2df43c01bd93cf978165c167dc7f982281a1501b7646c27e23abcba58cb6 WatchSource:0}: Error finding container acea2df43c01bd93cf978165c167dc7f982281a1501b7646c27e23abcba58cb6: Status 404 returned error can't find the container with id acea2df43c01bd93cf978165c167dc7f982281a1501b7646c27e23abcba58cb6
Nov 03 20:45:01.191427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:01.191346    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerStarted","Data":"5385284109c96161ce267d767e661541fc90ae0f8034cf8140a07b60d60620e4"}
Nov 03 20:45:01.191427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:01.191402    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerStarted","Data":"acea2df43c01bd93cf978165c167dc7f982281a1501b7646c27e23abcba58cb6"}
Nov 03 20:45:01.232210 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:01.232160    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" podStartSLOduration=1.232148205 podStartE2EDuration="1.232148205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:01.231473145 +0000 UTC m=+2443.440489148" watchObservedRunningTime="2026-11-03 20:45:01.232148205 +0000 UTC m=+2443.441164202"
Nov 03 20:45:02.197947 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:02.197910    5424 generic.go:356] "Generic (PLEG): container finished" podID="16d96139-652d-46cc-a98d-ac3d9301b890" containerID="5385284109c96161ce267d767e661541fc90ae0f8034cf8140a07b60d60620e4" exitCode=60
Nov 03 20:45:02.198342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:02.197956    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerDied","Data":"5385284109c96161ce267d767e661541fc90ae0f8034cf8140a07b60d60620e4"}
Nov 03 20:45:02.198342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:02.198179    5424 scope.go:122] "RemoveContainer" containerID="5385284109c96161ce267d767e661541fc90ae0f8034cf8140a07b60d60620e4"
Nov 03 20:45:03.205791 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:03.205742    5424 generic.go:356] "Generic (PLEG): container finished" podID="16d96139-652d-46cc-a98d-ac3d9301b890" containerID="49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288" exitCode=60
Nov 03 20:45:03.206275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:03.205832    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerDied","Data":"49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288"}
Nov 03 20:45:03.206275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:03.205912    5424 scope.go:122] "RemoveContainer" containerID="5385284109c96161ce267d767e661541fc90ae0f8034cf8140a07b60d60620e4"
Nov 03 20:45:03.206275 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:03.206148    5424 scope.go:122] "RemoveContainer" containerID="49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288"
Nov 03 20:45:03.206452 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:45:03.206410    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-kg9l6_odh-ai-gateway-infra(16d96139-652d-46cc-a98d-ac3d9301b890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" podUID="16d96139-652d-46cc-a98d-ac3d9301b890"
Nov 03 20:45:04.213506 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:04.213466    5424 scope.go:122] "RemoveContainer" containerID="49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288"
Nov 03 20:45:04.213936 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:45:04.213736    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-kg9l6_odh-ai-gateway-infra(16d96139-652d-46cc-a98d-ac3d9301b890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" podUID="16d96139-652d-46cc-a98d-ac3d9301b890"
Nov 03 20:45:06.074239 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.074201    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:45:06.074754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.074733    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:06.074805 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.074758    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:06.074905 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.074893    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ca4640-8625-4d65-97cd-5742801f27c0" containerName="istio-proxy"
Nov 03 20:45:06.079546 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.079505    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.095707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.095681    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:45:06.106949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.106920    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-229f0a\"/\"openshift-service-ca.crt\""
Nov 03 20:45:06.107067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.106949    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-229f0a\"/\"default-dockercfg-r96kp\""
Nov 03 20:45:06.107067 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.107003    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-229f0a\"/\"kube-root-ca.crt\""
Nov 03 20:45:06.107359 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.107342    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-229f0a\"/\"simulator-single-main-229f0a-kserve-self-signed-certs\""
Nov 03 20:45:06.234017 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.233989    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.234221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.234024    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.234221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.234156    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.234316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.234231    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvwc\" (UniqueName: \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.234316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.234285    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.234316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.234304    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.334979    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335019    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335055    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335075    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335129    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335149    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvwc\" (UniqueName: \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335446    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335514 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335464    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.335670 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.335584    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.337639 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.337621    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.337713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.337693    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.353968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.353937    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvwc\" (UniqueName: \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\") pod \"simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") " pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.394079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.394056    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:06.691827 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:06.691667    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:45:06.694636 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:45:06.694606    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb27e041_5618_4a69_83a5_d9e1dd93a440.slice/crio-5fbe848185163a475cce0659f8069e56a9faad9195e909b441af5b57f62f09ef WatchSource:0}: Error finding container 5fbe848185163a475cce0659f8069e56a9faad9195e909b441af5b57f62f09ef: Status 404 returned error can't find the container with id 5fbe848185163a475cce0659f8069e56a9faad9195e909b441af5b57f62f09ef
Nov 03 20:45:07.229198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:07.229161    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" event={"ID":"bb27e041-5618-4a69-83a5-d9e1dd93a440","Type":"ContainerStarted","Data":"b22a7a1557708b6f5a322e855b6dd6530e99995888ec415e9a4a24755713c8ee"}
Nov 03 20:45:07.229198 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:07.229206    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" event={"ID":"bb27e041-5618-4a69-83a5-d9e1dd93a440","Type":"ContainerStarted","Data":"5fbe848185163a475cce0659f8069e56a9faad9195e909b441af5b57f62f09ef"}
Nov 03 20:45:07.269412 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:07.269364    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" podStartSLOduration=1.26935145 podStartE2EDuration="1.26935145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.268086283 +0000 UTC m=+2449.477102291" watchObservedRunningTime="2026-11-03 20:45:07.26935145 +0000 UTC m=+2449.478367447"
Nov 03 20:45:07.394604 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:07.394567    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:12.284549 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:45:12.284494    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:45:15.912398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:15.912365    5424 scope.go:122] "RemoveContainer" containerID="49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288"
Nov 03 20:45:16.270958 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:16.270928    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerStarted","Data":"a0a2537cdd82518fe1f3fe3c8fd1bd4e3a391857bbbcde0ad5a994dac45bf374"}
Nov 03 20:45:16.394599 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:16.394567    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:16.406642 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:16.406619    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:17.277740 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.277705    5424 generic.go:356] "Generic (PLEG): container finished" podID="16d96139-652d-46cc-a98d-ac3d9301b890" containerID="a0a2537cdd82518fe1f3fe3c8fd1bd4e3a391857bbbcde0ad5a994dac45bf374" exitCode=60
Nov 03 20:45:17.278181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.277771    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerDied","Data":"a0a2537cdd82518fe1f3fe3c8fd1bd4e3a391857bbbcde0ad5a994dac45bf374"}
Nov 03 20:45:17.278181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.277843    5424 scope.go:122] "RemoveContainer" containerID="49fba9a69a40ff92895e6c60f4fc034696267b802e9ac603a77aa5ec85f63288"
Nov 03 20:45:17.278181 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.278158    5424 scope.go:122] "RemoveContainer" containerID="a0a2537cdd82518fe1f3fe3c8fd1bd4e3a391857bbbcde0ad5a994dac45bf374"
Nov 03 20:45:17.278382 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:45:17.278363    5424 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-kg9l6_odh-ai-gateway-infra(16d96139-652d-46cc-a98d-ac3d9301b890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" podUID="16d96139-652d-46cc-a98d-ac3d9301b890"
Nov 03 20:45:17.289621 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.289596    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:45:17.318146 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:17.318121    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:18.418855 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.418827    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:18.437927 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.437899    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\") pod \"16d96139-652d-46cc-a98d-ac3d9301b890\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") "
Nov 03 20:45:18.438076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.438055    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\" (UniqueName: \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\") pod \"16d96139-652d-46cc-a98d-ac3d9301b890\" (UID: \"16d96139-652d-46cc-a98d-ac3d9301b890\") "
Nov 03 20:45:18.438263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.438243    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca" pod "16d96139-652d-46cc-a98d-ac3d9301b890" (UID: "16d96139-652d-46cc-a98d-ac3d9301b890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:18.438366 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.438354    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d96139-652d-46cc-a98d-ac3d9301b890-maas-api-service-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:45:18.440165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.440142    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg" pod "16d96139-652d-46cc-a98d-ac3d9301b890" (UID: "16d96139-652d-46cc-a98d-ac3d9301b890"). InnerVolumeSpecName "kube-api-access-x8xfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:18.539165 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:18.539132    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xfg\" (UniqueName: \"kubernetes.io/projected/16d96139-652d-46cc-a98d-ac3d9301b890-kube-api-access-x8xfg\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:45:19.299393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.299354    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6" event={"ID":"16d96139-652d-46cc-a98d-ac3d9301b890","Type":"ContainerDied","Data":"acea2df43c01bd93cf978165c167dc7f982281a1501b7646c27e23abcba58cb6"}
Nov 03 20:45:19.299393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.299384    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"
Nov 03 20:45:19.299669 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.299430    5424 scope.go:122] "RemoveContainer" containerID="a0a2537cdd82518fe1f3fe3c8fd1bd4e3a391857bbbcde0ad5a994dac45bf374"
Nov 03 20:45:19.350911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.350877    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:19.363795 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.363765    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-kg9l6"]
Nov 03 20:45:19.918158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:19.918124    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" path="/var/lib/kubelet/pods/16d96139-652d-46cc-a98d-ac3d9301b890/volumes"
Nov 03 20:45:22.061459 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:22.061425    5424 scope.go:122] "RemoveContainer" containerID="c8990264cece55116888110a6c9b293ffd5b3d8f94dfefcfd93332b5d59d27da"
Nov 03 20:45:42.285583 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:45:42.285545    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:45:53.264338 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:45:53.264309    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:46:08.283728 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.283669    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerName="main" containerID="cri-o://b22a7a1557708b6f5a322e855b6dd6530e99995888ec415e9a4a24755713c8ee" gracePeriod=45
Nov 03 20:46:08.288244 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.288208    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.218:8000/ready\": EOF"
Nov 03 20:46:08.520931 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.520894    5424 generic.go:356] "Generic (PLEG): container finished" podID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerID="b22a7a1557708b6f5a322e855b6dd6530e99995888ec415e9a4a24755713c8ee" exitCode=0
Nov 03 20:46:08.521102 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.520966    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" event={"ID":"bb27e041-5618-4a69-83a5-d9e1dd93a440","Type":"ContainerDied","Data":"b22a7a1557708b6f5a322e855b6dd6530e99995888ec415e9a4a24755713c8ee"}
Nov 03 20:46:08.551664 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.551640    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:46:08.562757 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562735    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.562863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562811    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.562863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562846    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\" (UniqueName: \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.562957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562873    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.562957 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562935    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.563068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.562979    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\") pod \"bb27e041-5618-4a69-83a5-d9e1dd93a440\" (UID: \"bb27e041-5618-4a69-83a5-d9e1dd93a440\") "
Nov 03 20:46:08.563068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.563015    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:08.563177 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.563062    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:08.563263 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.563226    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:08.563330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.563311    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:08.563393 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.563332    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:08.565093 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.565072    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "kube-api-access-8mvwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:08.565191 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.565168    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:08.565241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.565188    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm" pod "bb27e041-5618-4a69-83a5-d9e1dd93a440" (UID: "bb27e041-5618-4a69-83a5-d9e1dd93a440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:08.664591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.664557    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvwc\" (UniqueName: \"kubernetes.io/projected/bb27e041-5618-4a69-83a5-d9e1dd93a440-kube-api-access-8mvwc\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:08.664591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.664587    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb27e041-5618-4a69-83a5-d9e1dd93a440-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:08.664591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.664597    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:08.664861 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:08.664604    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb27e041-5618-4a69-83a5-d9e1dd93a440-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:46:09.526897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.526862    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"
Nov 03 20:46:09.526897 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.526887    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc" event={"ID":"bb27e041-5618-4a69-83a5-d9e1dd93a440","Type":"ContainerDied","Data":"5fbe848185163a475cce0659f8069e56a9faad9195e909b441af5b57f62f09ef"}
Nov 03 20:46:09.527400 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.526935    5424 scope.go:122] "RemoveContainer" containerID="b22a7a1557708b6f5a322e855b6dd6530e99995888ec415e9a4a24755713c8ee"
Nov 03 20:46:09.578407 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.578382    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:46:09.591610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.591585    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-229f0a/simulator-single-main-229f0a-kserve-79bc6cdc7d-qw8kc"]
Nov 03 20:46:09.917160 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:46:09.917084    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" path="/var/lib/kubelet/pods/bb27e041-5618-4a69-83a5-d9e1dd93a440/volumes"
Nov 03 20:46:12.284907 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:46:12.284869    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:46:42.284112 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:46:42.284033    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:47:12.283831 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:47:12.283786    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:47:42.284789 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:47:42.284753    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:48:12.285972 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:48:12.285927    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:48:22.556192 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556156    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"]
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556486    5424 cpu_manager.go:414] "containerMap: removing container" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556497    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556507    5424 cpu_manager.go:414] "containerMap: removing container" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556511    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556520    5424 cpu_manager.go:414] "containerMap: removing container" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerName="main"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556539    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerName="main"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556595    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556608    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:22.556672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.556614    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb27e041-5618-4a69-83a5-d9e1dd93a440" containerName="main"
Nov 03 20:48:22.559737 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.559719    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.576913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.576889    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:48:22.589468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.589438    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:48:22.589468 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.589459    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:48:22.589718 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.589473    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:48:22.591063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.591038    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"]
Nov 03 20:48:22.643008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.642973    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.643172 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.643015    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.643219 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.643186    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjts\" (UniqueName: \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.743628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.743596    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjts\" (UniqueName: \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.743790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.743641    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.743790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.743657    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.744288 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.744256    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.746104 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.746081    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.762884 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.762856    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjts\" (UniqueName: \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\") pod \"maas-api-revoke-sub-d2bc93bc-jpf2m\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:22.872427 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:22.872342    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:23.171372 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:23.171347    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"]
Nov 03 20:48:23.174089 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:48:23.174054    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e5f5b2_6246_4ced_a791_8cd1c1ad6583.slice/crio-f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a WatchSource:0}: Error finding container f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a: Status 404 returned error can't find the container with id f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a
Nov 03 20:48:24.123132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:24.123091    5424 generic.go:356] "Generic (PLEG): container finished" podID="a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" containerID="944abc4a5e978586df9019aa62fa6ca9abd3c0ea04eac9af602bc0262041e488" exitCode=0
Nov 03 20:48:24.123523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:24.123179    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m" event={"ID":"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583","Type":"ContainerDied","Data":"944abc4a5e978586df9019aa62fa6ca9abd3c0ea04eac9af602bc0262041e488"}
Nov 03 20:48:24.123523 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:24.123240    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m" event={"ID":"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583","Type":"ContainerStarted","Data":"f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a"}
Nov 03 20:48:25.253949 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.253927    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:25.366807 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.366777    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\") pod \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") "
Nov 03 20:48:25.366973 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.366856    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\") pod \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") "
Nov 03 20:48:25.367046 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.367009    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\" (UniqueName: \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\") pod \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\" (UID: \"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583\") "
Nov 03 20:48:25.367151 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.367121    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca" pod "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" (UID: "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:25.367294 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.367280    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:48:25.368984 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.368963    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca" pod "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" (UID: "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:25.369082 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.369066    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts" pod "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" (UID: "a3e5f5b2-6246-4ced-a791-8cd1c1ad6583"). InnerVolumeSpecName "kube-api-access-qxjts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:25.468296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.468241    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:48:25.468296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:25.468262    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxjts\" (UniqueName: \"kubernetes.io/projected/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583-kube-api-access-qxjts\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:48:26.132729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:26.132699    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"
Nov 03 20:48:26.132729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:26.132712    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m" event={"ID":"a3e5f5b2-6246-4ced-a791-8cd1c1ad6583","Type":"ContainerDied","Data":"f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a"}
Nov 03 20:48:26.133066 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:26.132757    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f067bad7934fdaf466633349b69213213bfb20d5a67529676feec7cb2b08bc4a"
Nov 03 20:48:42.285840 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:48:42.285797    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:48:55.463109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463047    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463595    5424 cpu_manager.go:414] "containerMap: removing container" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463614    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463646    5424 cpu_manager.go:414] "containerMap: removing container" podUID="a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" containerName="revoke-keys"
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463657    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" containerName="revoke-keys"
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463762    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d96139-652d-46cc-a98d-ac3d9301b890" containerName="cleanup"
Nov 03 20:48:55.463910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.463775    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" containerName="revoke-keys"
Nov 03 20:48:55.467347 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.467320    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.483887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.483848    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0d89d1\"/\"openshift-service-ca.crt\""
Nov 03 20:48:55.483887 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.483848    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0d89d1\"/\"kube-root-ca.crt\""
Nov 03 20:48:55.496793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.496761    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0d89d1\"/\"simulator-single-main-0d89d1-kserve-self-signed-certs\""
Nov 03 20:48:55.496793 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.496779    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0d89d1\"/\"default-dockercfg-9zfj5\""
Nov 03 20:48:55.498413 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.498383    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:48:55.652925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.652886    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.652925 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.652929    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skg5m\" (UniqueName: \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.653206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.652961    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.653206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.653108    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.653206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.653189    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.653348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.653219    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754214 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754177    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754234    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754281    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754314    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skg5m\" (UniqueName: \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754360    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754422 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754397    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754739    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754772    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.754812 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.754790    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.756831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.756807    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.757038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.757016    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.773931 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.773902    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skg5m\" (UniqueName: \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\") pod \"simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") " pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:55.779745 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:55.779722    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:48:56.081808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:56.081685    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:48:56.084041 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:48:56.084014    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod025de069_137d_4dae_938f_4e1124ed8af7.slice/crio-aba2ddc487bab54dfd424295e8ef2fff2c0498676ffd978acbd545208175d074 WatchSource:0}: Error finding container aba2ddc487bab54dfd424295e8ef2fff2c0498676ffd978acbd545208175d074: Status 404 returned error can't find the container with id aba2ddc487bab54dfd424295e8ef2fff2c0498676ffd978acbd545208175d074
Nov 03 20:48:56.273013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:56.272971    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" event={"ID":"025de069-137d-4dae-938f-4e1124ed8af7","Type":"ContainerStarted","Data":"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"}
Nov 03 20:48:56.273013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:56.273013    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" event={"ID":"025de069-137d-4dae-938f-4e1124ed8af7","Type":"ContainerStarted","Data":"aba2ddc487bab54dfd424295e8ef2fff2c0498676ffd978acbd545208175d074"}
Nov 03 20:48:56.780517 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:48:56.780484    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:05.780348 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:05.780309    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:05.793556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:05.793511    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:05.845050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:05.844999    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" podStartSLOduration=10.844986195 podStartE2EDuration="10.844986195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:56.324051633 +0000 UTC m=+2678.533067629" watchObservedRunningTime="2026-11-03 20:49:05.844986195 +0000 UTC m=+2688.054002191"
Nov 03 20:49:06.329045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:06.329012    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:12.285171 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:49:12.285126    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:49:42.284511 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:49:42.284471    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:49:43.041672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:43.041634    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:49:58.061667 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.061619    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" podUID="025de069-137d-4dae-938f-4e1124ed8af7" containerName="main" containerID="cri-o://44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380" gracePeriod=45
Nov 03 20:49:58.333173 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.333149    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:58.382119 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382085    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\" (UniqueName: \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382146    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382289 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382243    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382319    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382355    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\" (UniqueName: \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382380    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:58.382432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382393    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\") pod \"025de069-137d-4dae-938f-4e1124ed8af7\" (UID: \"025de069-137d-4dae-938f-4e1124ed8af7\") "
Nov 03 20:49:58.382692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382582    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:58.382769 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382750    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.382831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382775    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.383011 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.382967    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:58.384723 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.384692    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "kube-api-access-skg5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:58.384975 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.384950    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:58.385073 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.385018    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm" pod "025de069-137d-4dae-938f-4e1124ed8af7" (UID: "025de069-137d-4dae-938f-4e1124ed8af7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:49:58.483541 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.483505    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.483713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.483550    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/025de069-137d-4dae-938f-4e1124ed8af7-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.483713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.483559    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/025de069-137d-4dae-938f-4e1124ed8af7-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.483713 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.483572    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skg5m\" (UniqueName: \"kubernetes.io/projected/025de069-137d-4dae-938f-4e1124ed8af7-kube-api-access-skg5m\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:49:58.559342 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.559300    5424 generic.go:356] "Generic (PLEG): container finished" podID="025de069-137d-4dae-938f-4e1124ed8af7" containerID="44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380" exitCode=0
Nov 03 20:49:58.559566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.559450    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" event={"ID":"025de069-137d-4dae-938f-4e1124ed8af7","Type":"ContainerDied","Data":"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"}
Nov 03 20:49:58.559566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.559487    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" event={"ID":"025de069-137d-4dae-938f-4e1124ed8af7","Type":"ContainerDied","Data":"aba2ddc487bab54dfd424295e8ef2fff2c0498676ffd978acbd545208175d074"}
Nov 03 20:49:58.559566 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.559508    5424 scope.go:122] "RemoveContainer" containerID="44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"
Nov 03 20:49:58.559748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.559707    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"
Nov 03 20:49:58.571000 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.570984    5424 scope.go:122] "RemoveContainer" containerID="44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"
Nov 03 20:49:58.571259 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:49:58.571240    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380\": container with ID starting with 44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380 not found: ID does not exist" containerID="44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"
Nov 03 20:49:58.571304 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.571267    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380"} err="failed to get container status \"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380\": rpc error: code = NotFound desc = could not find container \"44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380\": container with ID starting with 44c129daaa612ae94089bd28e41f118bd550b9790420a205e5d1f3b1ca187380 not found: ID does not exist"
Nov 03 20:49:58.617762 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.617720    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:49:58.626930 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:58.626905    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt"]
Nov 03 20:49:59.319024 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:59.318980    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-0d89d1/simulator-single-main-0d89d1-kserve-bf5c5b886-msnvt" podUID="025de069-137d-4dae-938f-4e1124ed8af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.220:8000/ready\": context deadline exceeded"
Nov 03 20:49:59.917613 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:49:59.917580    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025de069-137d-4dae-938f-4e1124ed8af7" path="/var/lib/kubelet/pods/025de069-137d-4dae-938f-4e1124ed8af7/volumes"
Nov 03 20:50:02.297454 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.297416    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:50:02.297854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.297774    5424 cpu_manager.go:414] "containerMap: removing container" podUID="025de069-137d-4dae-938f-4e1124ed8af7" containerName="main"
Nov 03 20:50:02.297854 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.297785    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025de069-137d-4dae-938f-4e1124ed8af7" containerName="main"
Nov 03 20:50:02.297933 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.297872    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="025de069-137d-4dae-938f-4e1124ed8af7" containerName="main"
Nov 03 20:50:02.302356 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.302333    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313765 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313736    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313806    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzxj\" (UniqueName: \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313855    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313875    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313894    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.313965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.313916    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.331170 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.331139    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:50:02.331340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.331254    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kschj\""
Nov 03 20:50:02.331340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.331286    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:50:02.331340 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.331293    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1b6f264d-kserve-self-signed-certs\""
Nov 03 20:50:02.332992 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.332963    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:50:02.415099 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415061    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415119    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzxj\" (UniqueName: \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415166    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415192    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415218    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415303 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415243    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415648    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415665    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.415798 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.415709    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.417496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.417462    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.417913 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.417897    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.434497 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.434474    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzxj\" (UniqueName: \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\") pod \"e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") " pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.614079 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.614002    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:02.914789 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:02.914758    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:50:02.916631 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:50:02.916592    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc3c068d_8dee_4aa5_9901_3174ac574c0f.slice/crio-c3af4be2c1e0f72332a5ad5f4805d6d4f39b6115c0880d1fe24fb8d4a55af97a WatchSource:0}: Error finding container c3af4be2c1e0f72332a5ad5f4805d6d4f39b6115c0880d1fe24fb8d4a55af97a: Status 404 returned error can't find the container with id c3af4be2c1e0f72332a5ad5f4805d6d4f39b6115c0880d1fe24fb8d4a55af97a
Nov 03 20:50:03.585008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:03.584970    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" event={"ID":"dc3c068d-8dee-4aa5-9901-3174ac574c0f","Type":"ContainerStarted","Data":"c13580e9efdc704e6bba021cd20ab6d27a76ec6b7b1ff02a8030566b3a8e4f31"}
Nov 03 20:50:03.585410 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:03.585019    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" event={"ID":"dc3c068d-8dee-4aa5-9901-3174ac574c0f","Type":"ContainerStarted","Data":"c3af4be2c1e0f72332a5ad5f4805d6d4f39b6115c0880d1fe24fb8d4a55af97a"}
Nov 03 20:50:03.614750 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:03.614722    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:03.628997 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:03.628954    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" podStartSLOduration=1.628940482 podStartE2EDuration="1.628940482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:03.623917341 +0000 UTC m=+2745.832933339" watchObservedRunningTime="2026-11-03 20:50:03.628940482 +0000 UTC m=+2745.837956470"
Nov 03 20:50:06.824720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:06.824683    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pd9ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:06.825147 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:06.824757    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pd9ml" podUID="064e0bc1-2bbe-4ba9-8be6-1d0074655e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:50:12.285094 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:50:12.285057    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:50:12.614638 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:12.614549    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:12.627325 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:12.627292    5424 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:12.638610 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:12.638586    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:50:42.286348 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:50:42.286306    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:50:51.769981 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:51.769926    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:50:56.889665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:56.889622    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mkxl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:56.890312 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:50:56.890275    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mkxl5" podUID="3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:06.795597 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:06.795542    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" podUID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" containerName="main" containerID="cri-o://c13580e9efdc704e6bba021cd20ab6d27a76ec6b7b1ff02a8030566b3a8e4f31" gracePeriod=45
Nov 03 20:51:06.929195 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:06.929157    5424 generic.go:356] "Generic (PLEG): container finished" podID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" containerID="c13580e9efdc704e6bba021cd20ab6d27a76ec6b7b1ff02a8030566b3a8e4f31" exitCode=0
Nov 03 20:51:06.929337 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:06.929222    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" event={"ID":"dc3c068d-8dee-4aa5-9901-3174ac574c0f","Type":"ContainerDied","Data":"c13580e9efdc704e6bba021cd20ab6d27a76ec6b7b1ff02a8030566b3a8e4f31"}
Nov 03 20:51:07.044409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.044384    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:51:07.122158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122090    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122158 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122134    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122168    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122342    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122394 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122379    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:07.122544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122399    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\" (UniqueName: \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122449    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\") pod \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\" (UID: \"dc3c068d-8dee-4aa5-9901-3174ac574c0f\") "
Nov 03 20:51:07.122544 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122397    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:07.122772 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122741    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:07.122865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122782    5424 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-home\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.122865 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.122798    5424 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tmp-dir\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.124556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.124501    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:07.124556 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.124499    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:07.124747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.124681    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj" pod "dc3c068d-8dee-4aa5-9901-3174ac574c0f" (UID: "dc3c068d-8dee-4aa5-9901-3174ac574c0f"). InnerVolumeSpecName "kube-api-access-cpzxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:07.224053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.224026    5424 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-model-cache\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.224053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.224050    5424 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3c068d-8dee-4aa5-9901-3174ac574c0f-tls-certs\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.224053 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.224059    5424 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3c068d-8dee-4aa5-9901-3174ac574c0f-dshm\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.224269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.224068    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpzxj\" (UniqueName: \"kubernetes.io/projected/dc3c068d-8dee-4aa5-9901-3174ac574c0f-kube-api-access-cpzxj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:51:07.934163 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.934134    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc" event={"ID":"dc3c068d-8dee-4aa5-9901-3174ac574c0f","Type":"ContainerDied","Data":"c3af4be2c1e0f72332a5ad5f4805d6d4f39b6115c0880d1fe24fb8d4a55af97a"}
Nov 03 20:51:07.934570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.934151    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"
Nov 03 20:51:07.934570 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.934176    5424 scope.go:122] "RemoveContainer" containerID="c13580e9efdc704e6bba021cd20ab6d27a76ec6b7b1ff02a8030566b3a8e4f31"
Nov 03 20:51:07.984330 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.984304    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:51:07.997206 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:07.997172    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1b6f264d-kserve-d986b764f-fj9pc"]
Nov 03 20:51:09.917339 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:09.917310    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" path="/var/lib/kubelet/pods/dc3c068d-8dee-4aa5-9901-3174ac574c0f/volumes"
Nov 03 20:51:12.284250 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:51:12.284216    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:51:42.285279 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:51:42.285243    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:51:44.421589 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.421563    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:51:44.421978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.421902    5424 cpu_manager.go:414] "containerMap: removing container" podUID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" containerName="main"
Nov 03 20:51:44.421978 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.421914    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" containerName="main"
Nov 03 20:51:44.422054 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.421990    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3c068d-8dee-4aa5-9901-3174ac574c0f" containerName="main"
Nov 03 20:51:44.426153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.426127    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.448853 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.448825    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:51:44.459784 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.459763    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e2966c90\""
Nov 03 20:51:44.459931 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.459914    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e2966c90-dockercfg-5szr6\""
Nov 03 20:51:44.530842 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.530809    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnf4\" (UniqueName: \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.531039 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.531005    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.631943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.631900    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.632128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.631952    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnf4\" (UniqueName: \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.632653 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.632633    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.651968 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.651942    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnf4\" (UniqueName: \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\") pod \"payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:44.739678 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:44.739650    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:45.036856 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:45.036794    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:51:45.039186 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:51:45.039153    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86d2bbf_6666_435e_9cc8_d8be5689c2fc.slice/crio-d12e25cbd10be2567d05851656943f6288e04413d8ffbecf133e764a15476bb1 WatchSource:0}: Error finding container d12e25cbd10be2567d05851656943f6288e04413d8ffbecf133e764a15476bb1: Status 404 returned error can't find the container with id d12e25cbd10be2567d05851656943f6288e04413d8ffbecf133e764a15476bb1
Nov 03 20:51:45.120557 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:45.120508    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" event={"ID":"c86d2bbf-6666-435e-9cc8-d8be5689c2fc","Type":"ContainerStarted","Data":"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"}
Nov 03 20:51:45.120729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:45.120579    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" event={"ID":"c86d2bbf-6666-435e-9cc8-d8be5689c2fc","Type":"ContainerStarted","Data":"d12e25cbd10be2567d05851656943f6288e04413d8ffbecf133e764a15476bb1"}
Nov 03 20:51:45.120729 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:45.120634    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:45.163230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:45.163177    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" podStartSLOduration=1.163163201 podStartE2EDuration="1.163163201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.161764517 +0000 UTC m=+2847.370780510" watchObservedRunningTime="2026-11-03 20:51:45.163163201 +0000 UTC m=+2847.372179192"
Nov 03 20:51:56.127586 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:56.127552    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:51:58.744182 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.744145    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"]
Nov 03 20:51:58.747919 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.747894    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.776651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.776621    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:51:58.776651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.776644    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:51:58.776651 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.776627    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:58.776892 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.776716    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v5zgv\""
Nov 03 20:51:58.779249 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.779226    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"]
Nov 03 20:51:58.870498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.870461    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.870695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.870597    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjvj\" (UniqueName: \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.870695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.870650    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.971552 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.971491    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjvj\" (UniqueName: \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.971754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.971566    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.971754 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.971617    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.972375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.972354    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.974240 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.974217    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:58.991045 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:58.991023    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjvj\" (UniqueName: \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\") pod \"maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:59.060243 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:59.060186    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:51:59.356594 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:51:59.356565    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"]
Nov 03 20:51:59.362560 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:51:59.362503    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1109ecb6_cf1e_44f9_a14e_189d8198ba83.slice/crio-40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9 WatchSource:0}: Error finding container 40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9: Status 404 returned error can't find the container with id 40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9
Nov 03 20:52:00.192235 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:00.192197    5424 generic.go:356] "Generic (PLEG): container finished" podID="1109ecb6-cf1e-44f9-a14e-189d8198ba83" containerID="b3e0b14d08d97f866b9295c56c624b63628156012a0270a7ed01c3327245dade" exitCode=0
Nov 03 20:52:00.192675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:00.192282    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz" event={"ID":"1109ecb6-cf1e-44f9-a14e-189d8198ba83","Type":"ContainerDied","Data":"b3e0b14d08d97f866b9295c56c624b63628156012a0270a7ed01c3327245dade"}
Nov 03 20:52:00.192675 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:00.192324    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz" event={"ID":"1109ecb6-cf1e-44f9-a14e-189d8198ba83","Type":"ContainerStarted","Data":"40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9"}
Nov 03 20:52:01.327665 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.327632    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:52:01.493428 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.493397    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\" (UniqueName: \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\") pod \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") "
Nov 03 20:52:01.493652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.493436    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\") pod \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") "
Nov 03 20:52:01.493652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.493485    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\") pod \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\" (UID: \"1109ecb6-cf1e-44f9-a14e-189d8198ba83\") "
Nov 03 20:52:01.494077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.493831    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca" pod "1109ecb6-cf1e-44f9-a14e-189d8198ba83" (UID: "1109ecb6-cf1e-44f9-a14e-189d8198ba83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:01.495845 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.495820    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj" pod "1109ecb6-cf1e-44f9-a14e-189d8198ba83" (UID: "1109ecb6-cf1e-44f9-a14e-189d8198ba83"). InnerVolumeSpecName "kube-api-access-hzjvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:01.495936 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.495848    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca" pod "1109ecb6-cf1e-44f9-a14e-189d8198ba83" (UID: "1109ecb6-cf1e-44f9-a14e-189d8198ba83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:01.594767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.594733    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjvj\" (UniqueName: \"kubernetes.io/projected/1109ecb6-cf1e-44f9-a14e-189d8198ba83-kube-api-access-hzjvj\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:52:01.594767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.594761    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:52:01.594767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:01.594771    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1109ecb6-cf1e-44f9-a14e-189d8198ba83-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:52:02.203321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:02.203239    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"
Nov 03 20:52:02.203321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:02.203253    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz" event={"ID":"1109ecb6-cf1e-44f9-a14e-189d8198ba83","Type":"ContainerDied","Data":"40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9"}
Nov 03 20:52:02.203321 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:02.203295    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e7ccac636fbe4bd4bf43662160bb1fe4df9ff9d4018ebe621209239614a0b9"
Nov 03 20:52:08.168971 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:08.168931    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:52:11.748129 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:11.748092    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"]
Nov 03 20:52:11.761911 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:11.761876    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2966c90-mbqcz"]
Nov 03 20:52:11.917687 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:11.917653    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1109ecb6-cf1e-44f9-a14e-189d8198ba83" path="/var/lib/kubelet/pods/1109ecb6-cf1e-44f9-a14e-189d8198ba83/volumes"
Nov 03 20:52:12.284352 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:52:12.284315    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:52:13.169912 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.169846    5424 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" podUID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" containerName="payload-pre-processing" containerID="cri-o://3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230" gracePeriod=25
Nov 03 20:52:13.420894 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.420841    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:52:13.505406 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.505373    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\" (UniqueName: \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\") pod \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") "
Nov 03 20:52:13.505598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.505418    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\") pod \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\" (UID: \"c86d2bbf-6666-435e-9cc8-d8be5689c2fc\") "
Nov 03 20:52:13.505829 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.505806    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume" pod "c86d2bbf-6666-435e-9cc8-d8be5689c2fc" (UID: "c86d2bbf-6666-435e-9cc8-d8be5689c2fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:13.507730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.507706    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4" pod "c86d2bbf-6666-435e-9cc8-d8be5689c2fc" (UID: "c86d2bbf-6666-435e-9cc8-d8be5689c2fc"). InnerVolumeSpecName "kube-api-access-gfnf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:13.607025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.606993    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnf4\" (UniqueName: \"kubernetes.io/projected/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-kube-api-access-gfnf4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:52:13.607025 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:13.607019    5424 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86d2bbf-6666-435e-9cc8-d8be5689c2fc-plugins-config-volume\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:52:14.262591 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.262554    5424 generic.go:356] "Generic (PLEG): container finished" podID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" containerID="3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230" exitCode=0
Nov 03 20:52:14.263086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.262592    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"
Nov 03 20:52:14.263086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.262633    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" event={"ID":"c86d2bbf-6666-435e-9cc8-d8be5689c2fc","Type":"ContainerDied","Data":"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"}
Nov 03 20:52:14.263086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.262694    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8" event={"ID":"c86d2bbf-6666-435e-9cc8-d8be5689c2fc","Type":"ContainerDied","Data":"d12e25cbd10be2567d05851656943f6288e04413d8ffbecf133e764a15476bb1"}
Nov 03 20:52:14.263086 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.262725    5424 scope.go:122] "RemoveContainer" containerID="3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"
Nov 03 20:52:14.271401 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.271381    5424 scope.go:122] "RemoveContainer" containerID="3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"
Nov 03 20:52:14.271663 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:52:14.271642    5424 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230\": container with ID starting with 3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230 not found: ID does not exist" containerID="3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"
Nov 03 20:52:14.271742 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.271674    5424 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230"} err="failed to get container status \"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230\": rpc error: code = NotFound desc = could not find container \"3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230\": container with ID starting with 3b5ac271aabfe70ef66b2a6d4f4dabf360c5a8432e2df5fa2200f31489f25230 not found: ID does not exist"
Nov 03 20:52:14.312934 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.312907    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:52:14.325598 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:14.325569    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2966c90-f5774bbff-v4hv8"]
Nov 03 20:52:15.918660 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:52:15.918624    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" path="/var/lib/kubelet/pods/c86d2bbf-6666-435e-9cc8-d8be5689c2fc/volumes"
Nov 03 20:52:42.288446 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:52:42.288361    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:53:12.285868 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:53:12.285828    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:53:26.028543 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:26.028495    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"]
Nov 03 20:53:26.040994 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:26.040968    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jpf2m"]
Nov 03 20:53:27.924364 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:27.924325    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e5f5b2-6246-4ced-a791-8cd1c1ad6583" path="/var/lib/kubelet/pods/a3e5f5b2-6246-4ced-a791-8cd1c1ad6583/volumes"
Nov 03 20:53:42.283968 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:53:42.283926    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:53:53.534501 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.534468    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"]
Nov 03 20:53:53.535152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.534988    5424 cpu_manager.go:414] "containerMap: removing container" podUID="1109ecb6-cf1e-44f9-a14e-189d8198ba83" containerName="revoke-keys"
Nov 03 20:53:53.535152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.535008    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1109ecb6-cf1e-44f9-a14e-189d8198ba83" containerName="revoke-keys"
Nov 03 20:53:53.535152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.535063    5424 cpu_manager.go:414] "containerMap: removing container" podUID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" containerName="payload-pre-processing"
Nov 03 20:53:53.535152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.535071    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" containerName="payload-pre-processing"
Nov 03 20:53:53.535152 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.535149    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86d2bbf-6666-435e-9cc8-d8be5689c2fc" containerName="payload-pre-processing"
Nov 03 20:53:53.535432 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.535162    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="1109ecb6-cf1e-44f9-a14e-189d8198ba83" containerName="revoke-keys"
Nov 03 20:53:53.538085 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.538062    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.558478 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.558446    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"]
Nov 03 20:53:53.568616 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.568592    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:53.568760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.568650    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:53:53.568760 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.568693    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:53:53.568840 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.568776    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v5zgv\""
Nov 03 20:53:53.668076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.668037    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.668076 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.668086    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slf6\" (UniqueName: \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.668310 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.668121    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.769061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.769021    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.769061 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.769070    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slf6\" (UniqueName: \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.769296 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.769226    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.769752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.769730    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.771672 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.771650    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.788252 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.788201    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slf6\" (UniqueName: \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\") pod \"maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:53.853008 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:53.852984    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:54.149736 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:54.149656    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"]
Nov 03 20:53:54.152876 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:53:54.152832    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d63a5f0_e315_4cba_8532_a9c96f4db9e7.slice/crio-d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306 WatchSource:0}: Error finding container d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306: Status 404 returned error can't find the container with id d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306
Nov 03 20:53:54.718212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:54.718121    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn" event={"ID":"8d63a5f0-e315-4cba-8532-a9c96f4db9e7","Type":"ContainerStarted","Data":"0d91563d36e8e89082691d67c094a21ab256d4cc845998982c8766a65bba4c03"}
Nov 03 20:53:54.718212 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:54.718207    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn" event={"ID":"8d63a5f0-e315-4cba-8532-a9c96f4db9e7","Type":"ContainerStarted","Data":"d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306"}
Nov 03 20:53:54.756733 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:54.756683    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn" podStartSLOduration=1.75667128 podStartE2EDuration="1.75667128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:54.756026637 +0000 UTC m=+2976.965042635" watchObservedRunningTime="2026-11-03 20:53:54.75667128 +0000 UTC m=+2976.965687276"
Nov 03 20:53:55.724038 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:55.724002    5424 generic.go:356] "Generic (PLEG): container finished" podID="8d63a5f0-e315-4cba-8532-a9c96f4db9e7" containerID="0d91563d36e8e89082691d67c094a21ab256d4cc845998982c8766a65bba4c03" exitCode=0
Nov 03 20:53:55.724463 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:55.724063    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn" event={"ID":"8d63a5f0-e315-4cba-8532-a9c96f4db9e7","Type":"ContainerDied","Data":"0d91563d36e8e89082691d67c094a21ab256d4cc845998982c8766a65bba4c03"}
Nov 03 20:53:56.855767 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.855740    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:56.892618 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.892590    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\") pod \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") "
Nov 03 20:53:56.892748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.892653    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\" (UniqueName: \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\") pod \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") "
Nov 03 20:53:56.892748 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.892685    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\") pod \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\" (UID: \"8d63a5f0-e315-4cba-8532-a9c96f4db9e7\") "
Nov 03 20:53:56.893071 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.893050    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca" pod "8d63a5f0-e315-4cba-8532-a9c96f4db9e7" (UID: "8d63a5f0-e315-4cba-8532-a9c96f4db9e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:56.894885 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.894859    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6" pod "8d63a5f0-e315-4cba-8532-a9c96f4db9e7" (UID: "8d63a5f0-e315-4cba-8532-a9c96f4db9e7"). InnerVolumeSpecName "kube-api-access-6slf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:56.894945 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.894866    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca" pod "8d63a5f0-e315-4cba-8532-a9c96f4db9e7" (UID: "8d63a5f0-e315-4cba-8532-a9c96f4db9e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:56.993830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.993749    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:53:56.993830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.993794    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slf6\" (UniqueName: \"kubernetes.io/projected/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-kube-api-access-6slf6\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:53:56.993830 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:56.993808    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d63a5f0-e315-4cba-8532-a9c96f4db9e7-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:53:57.734092 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:57.734064    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"
Nov 03 20:53:57.734276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:57.734098    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn" event={"ID":"8d63a5f0-e315-4cba-8532-a9c96f4db9e7","Type":"ContainerDied","Data":"d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306"}
Nov 03 20:53:57.734276 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:53:57.734145    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d61954eb29d9d46d5301531102ceaf5b6e093d8ef86ad22f075dc560cc7306"
Nov 03 20:54:12.285172 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:54:12.285138    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:54:12.836847 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:12.836811    5424 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"]
Nov 03 20:54:12.848979 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:12.848953    5424 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-d04674-nwxbn"]
Nov 03 20:54:13.917482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:13.917448    5424 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d63a5f0-e315-4cba-8532-a9c96f4db9e7" path="/var/lib/kubelet/pods/8d63a5f0-e315-4cba-8532-a9c96f4db9e7/volumes"
Nov 03 20:54:36.928264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.925885    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"]
Nov 03 20:54:36.928264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.926627    5424 cpu_manager.go:414] "containerMap: removing container" podUID="8d63a5f0-e315-4cba-8532-a9c96f4db9e7" containerName="revoke-keys"
Nov 03 20:54:36.928264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.926647    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d63a5f0-e315-4cba-8532-a9c96f4db9e7" containerName="revoke-keys"
Nov 03 20:54:36.928264 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.926778    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d63a5f0-e315-4cba-8532-a9c96f4db9e7" containerName="revoke-keys"
Nov 03 20:54:36.930779 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.930751    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:36.945309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.945283    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"]
Nov 03 20:54:36.956948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.956926    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:36.956948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.956925    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:36.957109 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.956952    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:36.958353 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:36.958329    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v5zgv\""
Nov 03 20:54:37.103747 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.103716    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfjz\" (UniqueName: \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.103915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.103783    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.103915 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.103859    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.204843 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.204765    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfjz\" (UniqueName: \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.205015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.204846    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.205015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.204879    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.205476 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.205452    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.207395 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.207366    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.224545 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.224503    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfjz\" (UniqueName: \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\") pod \"maas-api-revoke-sub-f03629db-s72q4\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.243890 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.243857    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:37.540209 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.540182    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"]
Nov 03 20:54:37.542073 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:54:37.542045    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b45b23_2f90_46f0_902d_a780ca11a8cd.slice/crio-79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead WatchSource:0}: Error finding container 79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead: Status 404 returned error can't find the container with id 79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead
Nov 03 20:54:37.938473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.938395    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4" event={"ID":"48b45b23-2f90-46f0-902d-a780ca11a8cd","Type":"ContainerStarted","Data":"2590c1958a8cb6f189d320d09b0f7698f313fbaae39381f1e4ad8dd8976a11d8"}
Nov 03 20:54:37.938473 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.938453    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4" event={"ID":"48b45b23-2f90-46f0-902d-a780ca11a8cd","Type":"ContainerStarted","Data":"79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead"}
Nov 03 20:54:37.977648 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:37.977600    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4" podStartSLOduration=1.977587816 podStartE2EDuration="1.977587816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:37.976132279 +0000 UTC m=+3020.185148276" watchObservedRunningTime="2026-11-03 20:54:37.977587816 +0000 UTC m=+3020.186603814"
Nov 03 20:54:38.944542 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:38.944496    5424 generic.go:356] "Generic (PLEG): container finished" podID="48b45b23-2f90-46f0-902d-a780ca11a8cd" containerID="2590c1958a8cb6f189d320d09b0f7698f313fbaae39381f1e4ad8dd8976a11d8" exitCode=0
Nov 03 20:54:38.944948 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:38.944578    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4" event={"ID":"48b45b23-2f90-46f0-902d-a780ca11a8cd","Type":"ContainerDied","Data":"2590c1958a8cb6f189d320d09b0f7698f313fbaae39381f1e4ad8dd8976a11d8"}
Nov 03 20:54:40.076628 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.076604    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:40.230720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.230642    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\") pod \"48b45b23-2f90-46f0-902d-a780ca11a8cd\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") "
Nov 03 20:54:40.230720 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.230714    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\") pod \"48b45b23-2f90-46f0-902d-a780ca11a8cd\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") "
Nov 03 20:54:40.230961 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.230802    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\" (UniqueName: \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\") pod \"48b45b23-2f90-46f0-902d-a780ca11a8cd\" (UID: \"48b45b23-2f90-46f0-902d-a780ca11a8cd\") "
Nov 03 20:54:40.231081 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.231055    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca" pod "48b45b23-2f90-46f0-902d-a780ca11a8cd" (UID: "48b45b23-2f90-46f0-902d-a780ca11a8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:40.232940 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.232903    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca" pod "48b45b23-2f90-46f0-902d-a780ca11a8cd" (UID: "48b45b23-2f90-46f0-902d-a780ca11a8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:40.233050 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.232990    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz" pod "48b45b23-2f90-46f0-902d-a780ca11a8cd" (UID: "48b45b23-2f90-46f0-902d-a780ca11a8cd"). InnerVolumeSpecName "kube-api-access-fkfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:40.332402 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.332373    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:40.332554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.332405    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfjz\" (UniqueName: \"kubernetes.io/projected/48b45b23-2f90-46f0-902d-a780ca11a8cd-kube-api-access-fkfjz\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:40.332554 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.332421    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b45b23-2f90-46f0-902d-a780ca11a8cd-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:40.954058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.954020    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4" event={"ID":"48b45b23-2f90-46f0-902d-a780ca11a8cd","Type":"ContainerDied","Data":"79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead"}
Nov 03 20:54:40.954058 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.954051    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s72q4"
Nov 03 20:54:40.954272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:40.954070    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b2cbb2eb5374317527dfcf74591ef45be6ba3250cafbc72b648e454edc5ead"
Nov 03 20:54:42.284414 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:54:42.284376    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:54:53.240730 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.240688    5424 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-lv28h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:54:53.243513 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.241284    5424 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-lv28h" podUID="a5c89cdf-dad9-4a02-8e82-347059ce2a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:53.926148 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.926120    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"]
Nov 03 20:54:53.926442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.926430    5424 cpu_manager.go:414] "containerMap: removing container" podUID="48b45b23-2f90-46f0-902d-a780ca11a8cd" containerName="revoke-keys"
Nov 03 20:54:53.926489 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.926443    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b45b23-2f90-46f0-902d-a780ca11a8cd" containerName="revoke-keys"
Nov 03 20:54:53.926540 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.926501    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b45b23-2f90-46f0-902d-a780ca11a8cd" containerName="revoke-keys"
Nov 03 20:54:53.929606 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.929579    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:53.946211 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.946182    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:53.958756 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.958720    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:53.958910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.958787    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:53.958910 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.958802    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v5zgv\""
Nov 03 20:54:53.960831 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.960809    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"]
Nov 03 20:54:53.975052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.975030    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckk4\" (UniqueName: \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:53.975188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.975076    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:53.975188 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:53.975168    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.075954 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.075925    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckk4\" (UniqueName: \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.076132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.075987    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.076132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.076041    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.076695 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.076676    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.078442 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.078420    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.095408 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.095388    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckk4\" (UniqueName: \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\") pod \"maas-api-revoke-sub-0165d7d2-g84hn\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.242132 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.242108    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:54.543221 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:54.543197    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"]
Nov 03 20:54:54.547456 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:54:54.547426    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39266d4_0959_4cf1_81b2_abe324c258e1.slice/crio-f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960 WatchSource:0}: Error finding container f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960: Status 404 returned error can't find the container with id f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960
Nov 03 20:54:55.020028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:55.019991    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn" event={"ID":"f39266d4-0959-4cf1-81b2-abe324c258e1","Type":"ContainerStarted","Data":"fda09d1503fea59847040d352fc9246c6276d5b833f1e7eb109aef3c82e56a23"}
Nov 03 20:54:55.020184 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:55.020054    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn" event={"ID":"f39266d4-0959-4cf1-81b2-abe324c258e1","Type":"ContainerStarted","Data":"f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960"}
Nov 03 20:54:55.058674 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:55.058624    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn" podStartSLOduration=2.058611785 podStartE2EDuration="2.058611785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:55.058085766 +0000 UTC m=+3037.267101763" watchObservedRunningTime="2026-11-03 20:54:55.058611785 +0000 UTC m=+3037.267627774"
Nov 03 20:54:56.026052 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:56.026017    5424 generic.go:356] "Generic (PLEG): container finished" podID="f39266d4-0959-4cf1-81b2-abe324c258e1" containerID="fda09d1503fea59847040d352fc9246c6276d5b833f1e7eb109aef3c82e56a23" exitCode=0
Nov 03 20:54:56.026430 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:56.026087    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn" event={"ID":"f39266d4-0959-4cf1-81b2-abe324c258e1","Type":"ContainerDied","Data":"fda09d1503fea59847040d352fc9246c6276d5b833f1e7eb109aef3c82e56a23"}
Nov 03 20:54:57.159539 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.159501    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:57.201030 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.201000    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\" (UniqueName: \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\") pod \"f39266d4-0959-4cf1-81b2-abe324c258e1\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") "
Nov 03 20:54:57.201153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.201079    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\") pod \"f39266d4-0959-4cf1-81b2-abe324c258e1\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") "
Nov 03 20:54:57.201153 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.201108    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\") pod \"f39266d4-0959-4cf1-81b2-abe324c258e1\" (UID: \"f39266d4-0959-4cf1-81b2-abe324c258e1\") "
Nov 03 20:54:57.201498 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.201470    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca" pod "f39266d4-0959-4cf1-81b2-abe324c258e1" (UID: "f39266d4-0959-4cf1-81b2-abe324c258e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:57.203371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.203341    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca" pod "f39266d4-0959-4cf1-81b2-abe324c258e1" (UID: "f39266d4-0959-4cf1-81b2-abe324c258e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:57.203481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.203464    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4" pod "f39266d4-0959-4cf1-81b2-abe324c258e1" (UID: "f39266d4-0959-4cf1-81b2-abe324c258e1"). InnerVolumeSpecName "kube-api-access-kckk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:57.302316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.302251    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckk4\" (UniqueName: \"kubernetes.io/projected/f39266d4-0959-4cf1-81b2-abe324c258e1-kube-api-access-kckk4\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:57.302316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.302276    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:57.302316 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:57.302285    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39266d4-0959-4cf1-81b2-abe324c258e1-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:54:58.037302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:58.037273    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn"
Nov 03 20:54:58.037302 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:58.037283    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-g84hn" event={"ID":"f39266d4-0959-4cf1-81b2-abe324c258e1","Type":"ContainerDied","Data":"f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960"}
Nov 03 20:54:58.037581 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:54:58.037348    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f28f6c82d86c6212b405409eb9d44b3686fcabd0e9e10abb5ca05fb2b11ae960"
Nov 03 20:55:04.938548 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.938482    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"]
Nov 03 20:55:04.939063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.938819    5424 cpu_manager.go:414] "containerMap: removing container" podUID="f39266d4-0959-4cf1-81b2-abe324c258e1" containerName="revoke-keys"
Nov 03 20:55:04.939063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.938830    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39266d4-0959-4cf1-81b2-abe324c258e1" containerName="revoke-keys"
Nov 03 20:55:04.939063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.938885    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39266d4-0959-4cf1-81b2-abe324c258e1" containerName="revoke-keys"
Nov 03 20:55:04.942136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.942110    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:04.970877 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.970850    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:04.971063 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.971047    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:55:04.971142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.971048    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:55:04.971142 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.971128    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v5zgv\""
Nov 03 20:55:04.972652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:04.972627    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"]
Nov 03 20:55:05.070108 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.070077    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.070317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.070173    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbk2f\" (UniqueName: \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.070317 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.070201    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.171290 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.171261    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.171449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.171304    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbk2f\" (UniqueName: \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.171449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.171324    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.171929 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.171906    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.173741 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.173723    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.191127 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.191065    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbk2f\" (UniqueName: \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\") pod \"maas-api-revoke-sub-6b19338f-zdc7j\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.255128 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.255106    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:05.552863 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:05.552838    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"]
Nov 03 20:55:05.555379 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:55:05.555336    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd910f6f7_532b_4a1a_b1f3_1260caa52ba8.slice/crio-4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555 WatchSource:0}: Error finding container 4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555: Status 404 returned error can't find the container with id 4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555
Nov 03 20:55:06.076652 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:06.076615    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j" event={"ID":"d910f6f7-532b-4a1a-b1f3-1260caa52ba8","Type":"ContainerStarted","Data":"d23be3b2aa80c35575576900ec46c1edb821177105f2de7c1b1a54c3651ca84a"}
Nov 03 20:55:06.077155 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:06.076666    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j" event={"ID":"d910f6f7-532b-4a1a-b1f3-1260caa52ba8","Type":"ContainerStarted","Data":"4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555"}
Nov 03 20:55:06.116309 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:06.116262    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j" podStartSLOduration=2.116248891 podStartE2EDuration="2.116248891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:06.116183097 +0000 UTC m=+3048.325199092" watchObservedRunningTime="2026-11-03 20:55:06.116248891 +0000 UTC m=+3048.325264888"
Nov 03 20:55:07.083077 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:07.083039    5424 generic.go:356] "Generic (PLEG): container finished" podID="d910f6f7-532b-4a1a-b1f3-1260caa52ba8" containerID="d23be3b2aa80c35575576900ec46c1edb821177105f2de7c1b1a54c3651ca84a" exitCode=0
Nov 03 20:55:07.083515 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:07.083119    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j" event={"ID":"d910f6f7-532b-4a1a-b1f3-1260caa52ba8","Type":"ContainerDied","Data":"d23be3b2aa80c35575576900ec46c1edb821177105f2de7c1b1a54c3651ca84a"}
Nov 03 20:55:08.215375 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.215348    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:08.297655 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.297622    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\" (UniqueName: \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\") pod \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") "
Nov 03 20:55:08.297836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.297670    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\") pod \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") "
Nov 03 20:55:08.297836 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.297697    5424 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\") pod \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\" (UID: \"d910f6f7-532b-4a1a-b1f3-1260caa52ba8\") "
Nov 03 20:55:08.298107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.298083    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca" pod "d910f6f7-532b-4a1a-b1f3-1260caa52ba8" (UID: "d910f6f7-532b-4a1a-b1f3-1260caa52ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:08.299943 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.299918    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f" pod "d910f6f7-532b-4a1a-b1f3-1260caa52ba8" (UID: "d910f6f7-532b-4a1a-b1f3-1260caa52ba8"). InnerVolumeSpecName "kube-api-access-kbk2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:08.300055 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.299988    5424 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca" pod "d910f6f7-532b-4a1a-b1f3-1260caa52ba8" (UID: "d910f6f7-532b-4a1a-b1f3-1260caa52ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:08.398269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.398214    5424 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbk2f\" (UniqueName: \"kubernetes.io/projected/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-kube-api-access-kbk2f\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:55:08.398269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.398239    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-xks-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:55:08.398269 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:08.398248    5424 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d910f6f7-532b-4a1a-b1f3-1260caa52ba8-maas-api-openshift-ca\") on node \"ip-10-0-136-244.ec2.internal\" DevicePath \"\""
Nov 03 20:55:09.093122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:09.093088    5424 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j"
Nov 03 20:55:09.093295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:09.093088    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-zdc7j" event={"ID":"d910f6f7-532b-4a1a-b1f3-1260caa52ba8","Type":"ContainerDied","Data":"4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555"}
Nov 03 20:55:09.093295 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:09.093212    5424 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0b906cb81374159e622f3e9dc323e90580ed683bec34360fe89ddf4b923555"
Nov 03 20:55:12.284776 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:55:12.284741    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:55:22.172037 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:22.172004    5424 scope.go:122] "RemoveContainer" containerID="944abc4a5e978586df9019aa62fa6ca9abd3c0ea04eac9af602bc0262041e488"
Nov 03 20:55:27.301345 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:27.301264    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s8phc_79d945a9-0182-4a44-9280-92c12cbaca8f/manager/0.log"
Nov 03 20:55:29.352449 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:29.352416    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-g84hn_f39266d4-0959-4cf1-81b2-abe324c258e1/revoke-keys/0.log"
Nov 03 20:55:29.900169 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:29.900142    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zdc7j_d910f6f7-532b-4a1a-b1f3-1260caa52ba8/revoke-keys/0.log"
Nov 03 20:55:30.090028 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:30.090002    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s72q4_48b45b23-2f90-46f0-902d-a780ca11a8cd/revoke-keys/0.log"
Nov 03 20:55:34.352587 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:34.352558    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-m28cr_3a807a22-e50c-4464-9b68-4ec5d33f499f/kube-auth-proxy/0.log"
Nov 03 20:55:35.509197 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:35.509161    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-lv28h_a5c89cdf-dad9-4a02-8e82-347059ce2a4e/istio-proxy/0.log"
Nov 03 20:55:35.715969 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:35.715920    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mkxl5_3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9/istio-proxy/0.log"
Nov 03 20:55:36.106712 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:36.106677    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-pd9ml_064e0bc1-2bbe-4ba9-8be6-1d0074655e00/istio-proxy/0.log"
Nov 03 20:55:42.284693 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:55:42.284660    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:55:49.376509 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:49.376478    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s8phc_79d945a9-0182-4a44-9280-92c12cbaca8f/manager/0.log"
Nov 03 20:55:51.122388 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:51.122355    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-g84hn_f39266d4-0959-4cf1-81b2-abe324c258e1/revoke-keys/0.log"
Nov 03 20:55:51.478122 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:51.478043    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-zdc7j_d910f6f7-532b-4a1a-b1f3-1260caa52ba8/revoke-keys/0.log"
Nov 03 20:55:51.654224 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:51.654199    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s72q4_48b45b23-2f90-46f0-902d-a780ca11a8cd/revoke-keys/0.log"
Nov 03 20:55:55.275241 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:55.275208    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-m28cr_3a807a22-e50c-4464-9b68-4ec5d33f499f/kube-auth-proxy/0.log"
Nov 03 20:55:56.439563 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:56.439514    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-lv28h_a5c89cdf-dad9-4a02-8e82-347059ce2a4e/istio-proxy/0.log"
Nov 03 20:55:56.637137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:56.637105    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mkxl5_3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9/istio-proxy/0.log"
Nov 03 20:55:56.993383 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:55:56.993354    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-pd9ml_064e0bc1-2bbe-4ba9-8be6-1d0074655e00/istio-proxy/0.log"
Nov 03 20:56:12.283963 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:56:12.283925    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:56:42.284342 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:56:42.284305    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:57:10.064036 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.063955    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fng2b/must-gather-6xgph"]
Nov 03 20:57:10.064433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.064276    5424 cpu_manager.go:414] "containerMap: removing container" podUID="d910f6f7-532b-4a1a-b1f3-1260caa52ba8" containerName="revoke-keys"
Nov 03 20:57:10.064433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.064287    5424 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d910f6f7-532b-4a1a-b1f3-1260caa52ba8" containerName="revoke-keys"
Nov 03 20:57:10.064433 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.064351    5424 memory_manager.go:365] "RemoveStaleState removing state" podUID="d910f6f7-532b-4a1a-b1f3-1260caa52ba8" containerName="revoke-keys"
Nov 03 20:57:10.067398 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.067382    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.083649 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.083626    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"openshift-service-ca.crt\""
Nov 03 20:57:10.096230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.096203    5424 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"kube-root-ca.crt\""
Nov 03 20:57:10.096381 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.096288    5424 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fng2b\"/\"default-dockercfg-6f4g7\""
Nov 03 20:57:10.098001 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.097981    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/must-gather-6xgph"]
Nov 03 20:57:10.205522 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.205494    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk9fp\" (UniqueName: \"kubernetes.io/projected/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-kube-api-access-pk9fp\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.205681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.205658    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-must-gather-output\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.307013 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.306986    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-must-gather-output\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.307145 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.307031    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk9fp\" (UniqueName: \"kubernetes.io/projected/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-kube-api-access-pk9fp\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.307313 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.307296    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-must-gather-output\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.326707 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.326653    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk9fp\" (UniqueName: \"kubernetes.io/projected/e9e20304-320a-4da2-9e3c-6c1b7c5935c2-kube-api-access-pk9fp\") pod \"must-gather-6xgph\" (UID: \"e9e20304-320a-4da2-9e3c-6c1b7c5935c2\") " pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.387059 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.387036    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/must-gather-6xgph"
Nov 03 20:57:10.683068 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.683044    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/must-gather-6xgph"]
Nov 03 20:57:10.684901 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:57:10.684873    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e20304_320a_4da2_9e3c_6c1b7c5935c2.slice/crio-2b93043d5506f054a30f1a157ac3046cdfb935ebf535ca3918a888a2027ce611 WatchSource:0}: Error finding container 2b93043d5506f054a30f1a157ac3046cdfb935ebf535ca3918a888a2027ce611: Status 404 returned error can't find the container with id 2b93043d5506f054a30f1a157ac3046cdfb935ebf535ca3918a888a2027ce611
Nov 03 20:57:10.686770 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:10.686754    5424 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:57:11.686808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:11.686711    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/must-gather-6xgph" event={"ID":"e9e20304-320a-4da2-9e3c-6c1b7c5935c2","Type":"ContainerStarted","Data":"98a161af5c2d81de3900a4e5ff14923a07cae9c0b365ad30572f433347fc3c79"}
Nov 03 20:57:11.686808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:11.686764    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/must-gather-6xgph" event={"ID":"e9e20304-320a-4da2-9e3c-6c1b7c5935c2","Type":"ContainerStarted","Data":"3cb2d264b01a0c21f55fa0681c0f980365613ca9b2c84a0a6d9b63a4dc885414"}
Nov 03 20:57:11.686808 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:11.686781    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/must-gather-6xgph" event={"ID":"e9e20304-320a-4da2-9e3c-6c1b7c5935c2","Type":"ContainerStarted","Data":"2b93043d5506f054a30f1a157ac3046cdfb935ebf535ca3918a888a2027ce611"}
Nov 03 20:57:12.294635 ip-10-0-136-244 kubenswrapper[5424]: E1103 20:57:12.294595    5424 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-blp4x" containerName="extractor"
Nov 03 20:57:14.189882 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:14.189851    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7mnpc_e97d10c4-05fb-460d-ba88-87646ce6c054/global-pull-secret-syncer/0.log"
Nov 03 20:57:15.399421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:15.399389    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jqq6k_2f983d9e-13b0-40a1-99a9-8e31927ae731/konnectivity-agent/0.log"
Nov 03 20:57:15.683246 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:15.683167    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-244.ec2.internal_7a01309248349c50ad695700872f4edd/haproxy/0.log"
Nov 03 20:57:23.628421 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.628354    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fng2b/must-gather-6xgph" podStartSLOduration=13.004517854 podStartE2EDuration="13.628338368s" totalImagesPullingTime="623.820514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:11.725648889 +0000 UTC m=+3173.934664885" watchObservedRunningTime="2026-11-03 20:57:23.628338368 +0000 UTC m=+3185.837354362"
Nov 03 20:57:23.629174 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.629152    5424 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"]
Nov 03 20:57:23.633839 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.633817    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.651817 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.651789    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"]
Nov 03 20:57:23.672558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.672026    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-lib-modules\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.672558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.672091    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-proc\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.672558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.672122    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd6hw\" (UniqueName: \"kubernetes.io/projected/8b81f601-5b69-43df-8d23-61f38fd0fa32-kube-api-access-rd6hw\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.672558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.672155    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-sys\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.672558 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.672186    5424 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-podres\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775321    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd6hw\" (UniqueName: \"kubernetes.io/projected/8b81f601-5b69-43df-8d23-61f38fd0fa32-kube-api-access-rd6hw\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775371 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775365    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-sys\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775385    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-podres\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775453    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-lib-modules\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775477    5424 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-proc\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775576    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-proc\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775681 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775632    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-sys\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775684    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-podres\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.775965 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.775689    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b81f601-5b69-43df-8d23-61f38fd0fa32-lib-modules\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.796324 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.796284    5424 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd6hw\" (UniqueName: \"kubernetes.io/projected/8b81f601-5b69-43df-8d23-61f38fd0fa32-kube-api-access-rd6hw\") pod \"perf-node-gather-daemonset-8vwg2\" (UID: \"8b81f601-5b69-43df-8d23-61f38fd0fa32\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:23.950759 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:23.950678    5424 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:24.261482 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.261452    5424 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"]
Nov 03 20:57:24.263754 ip-10-0-136-244 kubenswrapper[5424]: W1103 20:57:24.263726    5424 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b81f601_5b69_43df_8d23_61f38fd0fa32.slice/crio-329c8cf2d347f089a10257241e2c22c5b09d7a51c8b61d8eb3be04ae77ec76c8 WatchSource:0}: Error finding container 329c8cf2d347f089a10257241e2c22c5b09d7a51c8b61d8eb3be04ae77ec76c8: Status 404 returned error can't find the container with id 329c8cf2d347f089a10257241e2c22c5b09d7a51c8b61d8eb3be04ae77ec76c8
Nov 03 20:57:24.798136 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.798098    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2" event={"ID":"8b81f601-5b69-43df-8d23-61f38fd0fa32","Type":"ContainerStarted","Data":"4e10f03009b370db3417b1b9f3ae16eb7a0bed5099ab9f4a83a4b0f189cc99a4"}
Nov 03 20:57:24.798692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.798162    5424 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2" event={"ID":"8b81f601-5b69-43df-8d23-61f38fd0fa32","Type":"ContainerStarted","Data":"329c8cf2d347f089a10257241e2c22c5b09d7a51c8b61d8eb3be04ae77ec76c8"}
Nov 03 20:57:24.798692 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.798279    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:24.836790 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.836734    5424 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2" podStartSLOduration=1.836718749 podStartE2EDuration="1.836718749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:24.835842113 +0000 UTC m=+3187.044858109" watchObservedRunningTime="2026-11-03 20:57:24.836718749 +0000 UTC m=+3187.045734747"
Nov 03 20:57:24.855850 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:24.855821    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hcz9r_eb4c317d-9756-4fe6-b9c5-cd6468ca8e7f/monitoring-plugin/0.log"
Nov 03 20:57:25.323033 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:25.323002    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4cg4_77075119-7593-44da-9382-75e29ed85ea5/node-exporter/0.log"
Nov 03 20:57:25.439555 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:25.439487    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4cg4_77075119-7593-44da-9382-75e29ed85ea5/kube-rbac-proxy/0.log"
Nov 03 20:57:25.556781 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:25.556755    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h4cg4_77075119-7593-44da-9382-75e29ed85ea5/init-textfile/0.log"
Nov 03 20:57:26.363496 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.363465    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/prometheus/0.log"
Nov 03 20:57:26.480403 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.480373    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/config-reloader/0.log"
Nov 03 20:57:26.598143 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.598102    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/thanos-sidecar/0.log"
Nov 03 20:57:26.716048 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.715976    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/kube-rbac-proxy-web/0.log"
Nov 03 20:57:26.835016 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.834986    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/kube-rbac-proxy/0.log"
Nov 03 20:57:26.953848 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:26.953823    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/kube-rbac-proxy-thanos/0.log"
Nov 03 20:57:27.071603 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:27.071574    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8583e6da-46d9-4a06-ab84-7714426042cf/init-config-reloader/0.log"
Nov 03 20:57:27.199752 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:27.199726    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hqr7x_e5e5787e-ed9a-49db-9f07-9c44621d63d1/prometheus-operator/0.log"
Nov 03 20:57:27.315481 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:27.315439    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hqr7x_e5e5787e-ed9a-49db-9f07-9c44621d63d1/kube-rbac-proxy/0.log"
Nov 03 20:57:30.815107 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:30.815072    5424 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-8vwg2"
Nov 03 20:57:31.772137 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:31.772108    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5974f9cd-t9czm_0d6e6d6c-0149-4036-b9c3-093b2c6f417a/console/0.log"
Nov 03 20:57:35.288939 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:35.288911    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-469dq_7f95c475-6751-400c-b456-ef13c62c7a45/dns/0.log"
Nov 03 20:57:35.448717 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:35.448685    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-469dq_7f95c475-6751-400c-b456-ef13c62c7a45/kube-rbac-proxy/0.log"
Nov 03 20:57:36.111015 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:36.110981    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4jcr_4cbb7cff-dc60-4331-834b-416dd7d685a5/dns-node-resolver/0.log"
Nov 03 20:57:36.636272 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:36.636236    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-m28cr_3a807a22-e50c-4464-9b68-4ec5d33f499f/kube-auth-proxy/0.log"
Nov 03 20:57:37.006168 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:37.006129    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-lv28h_a5c89cdf-dad9-4a02-8e82-347059ce2a4e/istio-proxy/0.log"
Nov 03 20:57:37.071409 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:37.071381    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mkxl5_3637dc05-a96f-4dae-8ae4-dcd42dfd9ac9/istio-proxy/0.log"
Nov 03 20:57:37.203341 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:37.203310    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-pd9ml_064e0bc1-2bbe-4ba9-8be6-1d0074655e00/istio-proxy/0.log"
Nov 03 20:57:37.633572 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:37.633491    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t4x2d_7d9853ca-6edf-4ccb-b311-e2f57137b882/node-ca/0.log"
Nov 03 20:57:41.242230 ip-10-0-136-244 kubenswrapper[5424]: I1103 20:57:41.242195    5424 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-m28cr_3a807a22-e50c-4464-9b68-4ec5d33f499f/kube-auth-proxy/0.log"